[et_pb_section admin_label=”section” background_image=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/header_port_elizabeth_hd_darker.jpg” transparent_background=”off” background_color=”rgba(0,0,0,0)” allow_player_pause=”off” inner_shadow=”off” parallax=”off” parallax_method=”off” padding_mobile=”off” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” make_equal=”off” use_custom_gutter=”off” fullwidth=”off” specialty=”off” disabled=”off”][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”off” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ padding_mobile=”off” allow_player_pause=”off” parallax=”off” parallax_method=”on” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”on” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_3″][\/et_pb_column][et_pb_column type=”2_3″][et_pb_divider admin_label=”Divider” color=”#ffffff” height=”200″ divider_style=”solid” divider_position=”top” divider_weight=”1px” hide_on_mobile=”on” disabled=”off”] [\/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Sans||||” text_font_size=”72″ text_font_size_tablet=”52″ text_text_color=”#ffffff” text_line_height=”1.1em” use_border_color=”off” border_style=”solid” custom_margin=”0px||0px|” disabled=”off” text_font_size_phone=”42″ text_font_size_last_edited=”on|phone” border_color=”#ffffff” custom_margin_phone=”30px||30px|” custom_margin_last_edited=”on|phone”]<\/p>\n
Plotting\u00a0schools Port Elizabeth\u00a0using shell scripting<\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section admin_label=”Section” transparent_background=”off” background_color=”#f7f7f4″ allow_player_pause=”off” inner_shadow=”off” parallax=”off” parallax_method=”off” padding_mobile=”off” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” make_equal=”off” use_custom_gutter=”off” fullwidth=”off” specialty=”off” disabled=”off”][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”0px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Sans||||” text_font_size=”10em” text_text_color=”#323232″ text_line_height=”1.1em” use_border_color=”off” border_style=”solid” custom_margin=”0px||0px|” disabled=”off”] 27 [\/et_pb_text][et_pb_divider admin_label=”Divider” color=”#aeaeac” show_divider=”on” divider_style=”solid” divider_position=”top” divider_weight=”1px” hide_on_mobile=”off” custom_css_main_element=”width:20px;” disabled=”off”] [\/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Sans|on|||” text_font_size=”18″ text_text_color=”#363636″ use_border_color=”off” border_style=”solid” custom_margin=”-10px||0px|” disabled=”off”] SEPTEMBER, 2016 [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Sans||||” text_text_color=”#02b875″ use_border_color=”off” border_style=”solid” custom_margin=”20px|||” disabled=”off”]<\/p>\n
Scripting<\/a><\/p>\n South Africa<\/a><\/p>\n [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Sans||||” text_text_color=”#02b875″ use_border_color=”off” border_style=”solid” custom_margin=”20px|||” disabled=”off”] Reading time: ~15 min [\/et_pb_text][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”24″ text_text_color=”#363636″ text_line_height=”1.4em” use_border_color=”off” border_style=”solid” custom_margin=”10px||0px|” disabled=”off”]<\/p>\n Earlier this year I was in Port Elizabeth, South Africa to help the organization Ready4Life<\/a> with their Smart School Network project.<\/p>\n To gain an overview of all the schools that can be connected to the Smart School Network we wanted to plot\u00a0all the schools in Port Elizabeth and the surrounded townships onto Google Maps.<\/p>\n The script explained\u00a0in\u00a0this post can be found on the wiki<\/a>.<\/p>\n [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”80px|||” disabled=”off” inline_fonts=”Abel,Amatic SC,Arimo,Arvo,Bevan,Bitter,Black Ops One,Boogaloo,Bree Serif,Calligraffitti,Cantata One,Cardo,Changa One,Cherry Cream Soda,Chewy,Comfortaa,Coming Soon,Covered By Your Grace,Crafty Girls,Crete Round,Crimson Text,Cuprum,Dancing Script,Dosis,Droid Sans,Droid Serif,Francois One,Fredoka One,The Girl Next Door,Gloria Hallelujah,Happy Monkey,Indie Flower,Josefin Slab,Judson,Kreon,Lato,Lato Light,Lobster,Leckerli One,PT Serif”]<\/p>\n My South African IT colleague was able to find his way on the ‘net’ and found a list of all the schools in Port Elizabeth.\u00a0As you can see in the image the list of schools he found had an suburb for every school.<\/p>\n In this overview there was not an exact address or sometimes there was no suburb provided for a school. We could solve this challenge by manually looking up every school.<\/p>\n [\/et_pb_text][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/website_w_schools_port_elizabeth.jpg” alt=”Overview schools Port Elizabeth” title_text=”Overview schools Port Elizabeth” show_in_lightbox=”on” url=”http:\/\/www.schools4sa.co.za\/province\/eastern-cape\/port-elizabeth\/” url_new_window=”on” use_overlay=”off” animation=”off” sticky=”off” align=”center” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” custom_margin=”50px||50px|” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”||0px|” disabled=”off”]<\/p>\n Since the list consisted of 175 schools\u00a0manually looking up every school would take quite some time.<\/p>\n My South African friend tried really hard to look up every school and collect coordinates, but at the end of the day he only found the coordinates for a couple of schools.<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”50px|||” custom_padding_tablet=”17px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”4_4″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/google_maps_schools_port_elizabeth_1.jpg” alt=”Schools Port Elizabeth onto Google Maps” title_text=”Schools Port Elizabeth onto Google Maps” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”on” align=”left” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29969489565″] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”16″ text_text_color=”#363636″ text_line_height=”1em” use_border_color=”off” border_style=”solid” custom_margin=”20px||0px|” disabled=”off”] Schools Port Elizabeth plotted onto Google Maps [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”12″ text_text_color=”#363636″ text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”6px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Photograph by Christiaan Colen available on Flickr<\/a><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”0px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled_on=”off|off|off” disabled=”off”][et_pb_column type=”1_3″][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” disabled_on=”off|off|off” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off”]<\/p>\n The next morning I thought we can do this easier, time to create a shell script for it. We copied all the information from the webpage into Excel and with the help of some Excel formulas created an easy overview of all the schools.<\/p>\n This overview could then be exported as a CSV file that used the ‘;’ (semicolon) sign as a separator to separate the data of the different fields such as the name of the school, suburb\u00a0or telephone number.<\/p>\n Exporting all the data to a CSV file made it easy to use it in Linux and write a script to process the data. Less then an hour of scripting resulted in the script that’s explained further in this post. It’s not that ‘fancy’ but it gets the job done with just a couple lines of code.<\/p>\n [\/et_pb_text][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/screenshots_csv_schools_port_elizabeth.jpg” alt=”CSV file with schools Port Elizabeth” title_text=”CSV file with schools Port Elizabeth” show_in_lightbox=”on” url_new_window=”off” use_overlay=”off” animation=”off” sticky=”off” align=”center” force_fullwidth=”off” always_center_on_mobile=”on” disabled_on=”off|off|off” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px||0px|” disabled=”off”]<\/p>\n I’m going to use the Google Maps Places API to retrieve coordinates for a school. For those who don’t know what an API is, in short ‘an API is the messenger that runs and delivers your request to the provider you\u2019re requesting it from, and then delivers the response back to you’. For more information see this visual explanation<\/a>.<\/p>\n To be more accurate I’m using\u00a0the Google Maps Text Search Service<\/a>.\u00a0A short description from the Google Maps API documentation:<\/p>\n “The Google Places API Text Search Service is a web service that returns information about a set of places based on a string \u2014 for example “pizza in New York” or “shoe stores near Ottawa”. The service responds with a list of places matching the text string and any location bias that has been set.<\/em>”<\/p>\n This means that I’m asking Google Maps for coordinates (latitude, longitude) given a point of interest. In this case it’s based on the school’s name and place, for example Philip Nikiwe P School Port Elizabeth needs to result into 33.8974800 , 25.5875800 (coordinates).<\/p>\n Before using this Google Maps service you need to retrieve an API key first, to retrieve an API key have a look at the documentation<\/a>.<\/p>\n After saving all the retrieved coordinates, names of the schools\u00a0and addresses into a new CSV file we could upload the data to Google’s Fusion Tables<\/a>\u00a0to automatically plot it onto Google Maps. The result can be viewed here<\/a>.<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”50px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”4_4″][et_pb_divider admin_label=”Divider” color=”#aeaeac” show_divider=”on” divider_style=”solid” divider_position=”top” divider_weight=”1px” hide_on_mobile=”off” disabled=”off”] [\/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” max_width=”900px” text_font=”PT Serif||||” text_font_size=”48″ text_font_size_tablet=”38″ text_text_color=”#363636″ text_line_height=”1.3em” use_border_color=”off” border_style=”solid” custom_margin=”16px||30px|” disabled=”off”] Nothing is impossible, the word itself says ‘I’m possible’! –\u00a0Audrey Hepburn [\/et_pb_text][et_pb_divider admin_label=”Divider” color=”#aeaeac” show_divider=”on” divider_style=”solid” divider_position=”top” divider_weight=”1px” hide_on_mobile=”off” disabled=”off”] [\/et_pb_divider][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”16px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_3″][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off”]<\/p>\n Now it’s time for the technical part! Whenever you write\u00a0a\u00a0shell script you need to start with ‘#!\/bin\/bash<\/strong>‘ so the computer knows what interpreter (language) is going to be used to process the instructions.<\/p>\n The ‘#<\/strong>‘ (hash) sign in a shell script will allow you to write comments, like reminders.<\/p>\n Lets start with the first line of code ‘the while loop’. The basic principle of the while loop in this case is while reading a file do something.<\/p>\n [\/et_pb_text][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/shell_script_schools_port_elizabeth_2.jpg” alt=”Shell script point of interest to coordinates” title_text=”Shell script point of interest to coordinates” show_in_lightbox=”on” url_new_window=”off” use_overlay=”off” animation=”off” sticky=”off” align=”center” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” custom_margin=”5%||5%|” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px||0px|” disabled=”off”]<\/p>\n For this script it means while readin<\/span>g a line<\/span> from the file schools1.csv do read all the input into different variables<\/span> that are separated by the ‘;’ sign to store the data.<\/p>\n This makes it possible to read a line and store the name of a school into the variable called ‘name’ or to read the telephone number and store it into the variable ‘tel’.<\/p>\n while IFS=”;” read -r name tel phase suburb special<\/strong><\/p>\n do<\/strong><\/p>\n something<\/strong><\/p>\n done < “schools1.csv”<\/strong><\/p>\n [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Onto the next line:<\/p>\n poi=$(echo $name | sed “s| |+|g” | sed “s|To Be Updated||g” | sed “s|Port Elizabeth||g” | sed ‘s|$|+Port+Elizabeth|’)<\/strong><\/p>\n Here we create a variable called ‘poi<\/span>‘ (short for point of interest). The data in this variable will be used in a next step to request data via the Google Maps API.\u00a0We need to prepare the data a bit so it has always the same structure, otherwise it’s possible that some of the requests won’t give a correct result due to misunderstanding the data when performing a request.<\/p>\n The first thing we do is ‘echo $name<\/span>‘, which will put the name of the school into the variable ‘poi<\/span>‘.\u00a0The data could sometimes be containing more then only the name of a school. For this reason\u00a0we are going to replace or remove certain text strings with the ‘sed<\/span>‘ command so the data has always a certain structure.<\/p>\n The sed command ‘sed “s| |+|g”<\/span>‘ will replace all empty spaces (also called white space) with a ‘+’ sign, this is necessary for the API request. By replacing empty spaces with a ‘+’ sign the Google Maps API will know that there needs to be an empty space when proccesing the request.<\/p>\n Going back to the school example:\u00a0Philip Nikiwe P School Port Elizabeth will translate into\u00a0Philip+Nikiwe+P+School+Port+Elizabeth. \u00a0When you look at an URL of a website you will never see an empty space in the URL (it’s not allowed), normally it’s converted to a ‘+’ (plus) or ‘-‘ (hyphen) or ‘_’ (underscore) sign.<\/p>\n The next two commands (sed “s|To Be Updated||g” | sed “s|Port Elizabeth||g”)<\/span> will replace ‘To be Updated’ and ‘Port Elizabeth’ with nothing, it’s removing it if it’s a part of the data.<\/p>\n The last command ‘sed “s|$|+Port+Elizabeth|”<\/span>‘\u00a0will append ‘+Port+Elizabeth’ to all the data. By doing this\u00a0Philip+Nikiwe+P+School will result into\u00a0Philip+Nikiwe+P+School+Port+Elizabeth.<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”50px|||” custom_padding_tablet=”17px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”4_4″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_7.jpg” alt=”The different variable in the shell script” title_text=”The different variable in the shell script” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”on” align=”left” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29854858262″] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”16″ text_text_color=”#363636″ text_line_height=”1em” use_border_color=”off” border_style=”solid” custom_margin=”20px||0px|” disabled=”off” border_color=”#ffffff”]<\/p>\n The different variables in the shell script.<\/p>\n [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”12″ text_text_color=”#363636″ text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”6px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Photograph by Christiaan Colen available on Flickr<\/a><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”0px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled_on=”off|off|off” disabled=”off”][et_pb_column type=”1_3″][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Next up doing the actual request to the Google Maps API. This is done by the next line in the script:<\/p>\n request=$(curl –silent “https:\/\/maps.googleapis.com\/maps\/api\/place\/textsearch\/xml?query=”$poi”&key=insertyourkey”)<\/strong><\/p>\n Here we create a variable called ‘request<\/span>‘. We use the Linux ‘curl<\/span>‘ program to do a web request (ask a question). You can compare it to you typing the web request in the URL field of your browser. The ‘–silent<\/span>‘ will suppress output to the command line, this means the generated output won’t be shown on the screen.<\/p>\n Next is the Google Maps API URL address. The full request will look something like:<\/p>\n request=$(curl –silent “https:\/\/maps.googleapis.com\/maps\/api\/place\/textsearch\/xml?query=Philip+Nikiwe+P+School+Port+Elizabeth&key=APIKEY1234”)<\/strong><\/p>\n As you can see after textsearch we define that the response needs to be in XML format. For those reading that don’t know what XML means the Wikipedia<\/a> definition:<\/p>\n “Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable.<\/em>”<\/p>\n To make it more visual in the image below the XML output of the request for\u00a0Philip Nikiwe P School Port Elizabeth.<\/p>\n [\/et_pb_text][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/xml_response_google_maps.jpg” alt=”XML example Google Maps API output” title_text=”XML example Google Maps API output” show_in_lightbox=”on” url_new_window=”off” use_overlay=”off” animation=”off” sticky=”off” align=”center” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” disabled=”off”] [\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”on” gutter_width=”2″ custom_padding=”50px|||” custom_padding_tablet=”30px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_2″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_4-3.jpg” alt=”Shell script poi to coordinates” title_text=”Shell script poi to coordinates” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”off” align=”left” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29854860342″] [\/et_pb_image][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_1-10.jpg” alt=”Shell script poi to coordinates” title_text=”Shell script poi to coordinates” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”off” align=”left” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” custom_margin=”34px|||” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29675236050″] [\/et_pb_image][\/et_pb_column][et_pb_column type=”1_2″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_2-2.jpg” alt=”Shell script poi to coordinates” title_text=”Shell script poi to coordinates” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”off” align=”left” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29854861042″] [\/et_pb_image][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_3-2.jpg” alt=”Shell script poi to coordinates” title_text=”Shell script poi to coordinates” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”off” align=”left” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” custom_margin=”34px|||” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29854860712″] [\/et_pb_image][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”17px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_3″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_5-2.jpg” alt=”Schools Port Elizabeth onto Google Maps” title_text=”Schools Port Elizabeth onto Google Maps” show_in_lightbox=”on” url_new_window=”off” use_overlay=”off” sticky=”off” align=”left” max_width=”140px” force_fullwidth=”off” always_center_on_mobile=”off” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” custom_margin=”36px||0px|” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”140px” text_font=”PT Serif||on||” text_font_size=”16″ text_text_color=”#363636″ text_line_height=”1.1em” use_border_color=”off” border_style=”solid” custom_margin=”16px||0px|” disabled=”off”] Schools Port Elizabeth plotted onto Google Maps. [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”140px” text_font=”PT Serif||on||” text_font_size=”12″ text_text_color=”#363636″ text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”6px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Photographs by Christiaan Colen available on\u00a0Flickr<\/span><\/a><\/p>\n [\/et_pb_text][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px||0px|” disabled=”off” border_color=”#ffffff”]<\/p>\n Next step:\u00a0formatted=$(echo $request | grep -o -P ‘(?<=<formatted_address>).*(?=<\/formatted_address>)’)<\/strong><\/p>\n Here we create a variable called ‘formatted<\/span>‘. The purpose of this variable is to get the ‘formatted address’ Google Maps provided. According to the Google documentation:<\/p>\n “formatted_address is a string containing the human-readable address of this location. Often this address is equivalent to the “postal address,” which sometimes differs from country to country.<\/em>”<\/p>\n How do we get the formatted address?<\/p>\n It’s quite easy since the output we requested is XML we only have to get the values between the formatted_address tags. For the variable we first\u00a0‘echo<\/span>‘\u00a0(‘print’)\u00a0the output of the web request and then use ‘grep<\/span>‘ to get all the data between the ‘<formatted_address><\/span>‘ and ‘<\/formatted_address><\/span>‘ strings. ‘grep -o<\/span>‘ means that it will only take the data that is asked, no data surrounding the found data. That’s it.<\/p>\n Back to\u00a0the\u00a0Philip Nikiwe P School Port Elizabeth example, the output of the above command will result into ‘New Brighton 1, Ibahyi, 6200, South Africa’.<\/p>\n The same is done for the latitude (<lat>) and longtitude (<lng>) values.<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”50px|||” custom_padding_tablet=”17px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”4_4″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/09\/blogpost_shell_script_schools_port_elizabeth_6-1.jpg” alt=”Shell script school’s name to coordinates” title_text=”Shell script school’s name to coordinates” show_in_lightbox=”off” url_new_window=”on” use_overlay=”off” animation=”fade_in” sticky=”on” align=”left” force_fullwidth=”off” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” url=”https:\/\/www.flickr.com\/photos\/132889348@N07\/29675235080″] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”16″ text_text_color=”#363636″ text_line_height=”1em” use_border_color=”off” border_style=”solid” custom_margin=”20px||0px|” disabled=”off” border_color=”#ffffff”]<\/p>\n Getting the latitude and longitude values.<\/p>\n [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” text_font=”PT Serif||on||” text_font_size=”12″ text_text_color=”#363636″ text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”6px|||” disabled=”off” border_color=”#ffffff”]<\/p>\n Photograph by Christiaan Colen available on Flickr<\/a><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ custom_padding=”0px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled_on=”off|off|off” disabled=”off”][et_pb_column type=”1_3″][\/et_pb_column][et_pb_column type=”2_3″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” max_width=”620px” text_font=”PT Serif||||” text_font_size=”20″ text_text_color=”#363636″ text_line_height=”1.5em” use_border_color=”off” border_style=”solid” custom_margin=”30px||0px|” disabled=”off” border_color=”#ffffff”]<\/p>\n Once we have these three values the next step is to ‘echo<\/span>‘ (‘print’) all the data to the screen so you can see it while the script is running.<\/p>\n After generating the output to the screen the script will go to ‘sleep<\/span>‘ for 5 seconds. This means that no command is executed for 5 seconds and had to be done because I got problems with asking the Google Maps API too much too fast.<\/p>\n The last step:<\/p>\n echo $name”;”$formatted”;”$lat”;”$long >> schools_w_coordinates.csv<\/strong><\/p>\n This will store all the data into the file ‘schools_w_coordinates.csv<\/span>‘ separated by\u00a0‘;’ sign.<\/p>\n In case of the Philip Nikiwe P School example it will look like:<\/p>\n echo\u00a0Philip Nikiwe P School;New Brighton 1, Ibahyi, 6200, South Africa;-33.8974800;25.5875800<\/strong><\/p>\n The ‘>><\/span>‘\u00a0(two times greater than sign) is used to append the data to the file every time an entry in the while loop is done. If you use one ‘><\/span>‘ \u00a0sign then it will create a new file for every entry that is processed which means (since the filename is the same for every entry) it would overwrite the existing file.<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section admin_label=”Section” transparent_background=”off” allow_player_pause=”off” inner_shadow=”off” parallax=”off” parallax_method=”on” padding_mobile=”off” make_fullwidth=”off” use_custom_width=”off” width_unit=”off” custom_width_px=”1080px” custom_width_percent=”80%” make_equal=”off” use_custom_gutter=”off” fullwidth=”off” specialty=”off” disabled=”off”][et_pb_row admin_label=”Row” make_fullwidth=”off” use_custom_width=”on” width_unit=”on” custom_width_px=”780px” custom_width_percent=”80%” use_custom_gutter=”off” gutter_width=”3″ padding_mobile=”off” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”on” parallax_3=”off” parallax_method_3=”on” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”4_4″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Serif||||” text_font_size=”36″ text_font_size_tablet=”26″ text_text_color=”#363636″ text_line_height=”1.4em” use_border_color=”off” border_style=”solid” custom_margin=”||70px|” disabled=”off”] Music videos of the day [\/et_pb_text][et_pb_video_slider admin_label=”Video Slider” show_image_overlay=”hide” show_arrows=”on” show_thumbnails=”off” controls_color=”dark” disabled=”off”] [et_pb_video_slider_item src=”https:\/\/www.youtube.com\/watch?v=-JkWUfUM_-o” image_src=”\/\/i.ytimg.com\/vi\/-JkWUfUM_-o\/hqdefault.jpg” background_layout=”dark” disabled=”off” show_image_overlay=”hide”] [\/et_pb_video_slider_item][et_pb_video_slider_item src=”https:\/\/www.youtube.com\/watch?v=Cwm6ZmRZP1M” image_src=”\/\/i.ytimg.com\/vi\/Cwm6ZmRZP1M\/hqdefault.jpg” background_layout=”dark” disabled=”off” show_image_overlay=”hide”] [\/et_pb_video_slider_item][et_pb_video_slider_item src=”https:\/\/www.youtube.com\/watch?v=yvXmMOEuWwE” image_src=”\/\/i.ytimg.com\/vi\/yvXmMOEuWwE\/hqdefault.jpg” background_layout=”dark” disabled=”off” show_image_overlay=”hide”] [\/et_pb_video_slider_item][et_pb_video_slider_item src=”https:\/\/www.youtube.com\/watch?v=pgQQ7k30Ti4″ image_src=”\/\/i.ytimg.com\/vi\/pgQQ7k30Ti4\/hqdefault.jpg” background_layout=”dark” disabled=”off” show_image_overlay=”hide”][\/et_pb_video_slider_item] [\/et_pb_video_slider][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section admin_label=”Section” transparent_background=”off” background_color=”#222222″ allow_player_pause=”off” inner_shadow=”off” parallax=”off” parallax_method=”off” custom_padding=”0px||0px|” padding_mobile=”on” make_fullwidth=”off” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” make_equal=”off” use_custom_gutter=”off” fullwidth=”off” specialty=”off” disabled=”off”][et_pb_row admin_label=”Row” make_fullwidth=”on” use_custom_width=”off” width_unit=”on” custom_width_px=”1080px” custom_width_percent=”80%” use_custom_gutter=”on” gutter_width=”2″ custom_padding=”40px|||” padding_mobile=”on” allow_player_pause=”off” parallax=”off” parallax_method=”off” make_equal=”off” column_padding_mobile=”on” parallax_1=”off” parallax_method_1=”off” parallax_2=”off” parallax_method_2=”off” parallax_3=”off” parallax_method_3=”off” parallax_4=”off” parallax_method_4=”on” disabled=”off”][et_pb_column type=”1_3″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2016\/06\/flickr_api_code3_hd.jpg” show_in_lightbox=”off” url=”https:\/\/blog.christiaan008.com\/2016\/07\/12\/retrieving-flickr-photo-views-using-shell-scripting\/” url_new_window=”off” use_overlay=”off” animation=”fade_in” sticky=”off” align=”center” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_style=”solid” disabled=”off” alt=”Retrieving Flickr photo views using shell scripting” title_text=”Retrieving Flickr photo views using shell scripting”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans|on|||” text_font_size=”12″ text_text_color=”#888888″ text_letter_spacing=”2px” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off”] SCRIPTING<\/a> [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans||||” text_font_size=”24″ text_text_color=”#ffffff” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” disabled=”off” border_color=”#ffffff”] Retrieving\u00a0Flickr photo <\/a>views using shell scripting<\/a><\/span> [\/et_pb_text][\/et_pb_column][et_pb_column type=”1_3″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2015\/11\/shell_script1_high.jpg” show_in_lightbox=”off” url=”https:\/\/blog.christiaan008.com\/2015\/11\/08\/dynamically-splitting-output-into-files-while-processing\/” url_new_window=”off” use_overlay=”off” animation=”fade_in” sticky=”on” align=”center” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans|on|||” text_font_size=”12″ text_text_color=”#888888″ text_letter_spacing=”2px” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off”] SCRIPTING<\/a> [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans||||” text_font_size=”24″ text_text_color=”#ffffff” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” disabled=”off”]<\/p>\n Dynamically splitting output into <\/span><\/a> [\/et_pb_text][\/et_pb_column][et_pb_column type=”1_3″][et_pb_image admin_label=”Image” src=”https:\/\/blog.christiaan008.com\/wp-content\/uploads\/sites\/2\/2015\/01\/cover.jpg” show_in_lightbox=”off” url=”https:\/\/blog.christiaan008.com\/2015\/01\/16\/building-a-pole-for-the-wifi-project\/” url_new_window=”off” use_overlay=”off” animation=”fade_in” sticky=”off” align=”center” force_fullwidth=”on” always_center_on_mobile=”on” use_border_color=”off” border_color=”#ffffff” border_width=”1px” border_style=”solid” disabled=”off”] [\/et_pb_image][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans|on|||” text_font_size=”12″ text_text_color=”#888888″ text_letter_spacing=”2px” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” custom_margin=”30px|||” disabled=”off”] SOUTH AFRICA<\/a> [\/et_pb_text][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”center” text_font=”PT Sans||||” text_font_size=”24″ text_text_color=”#ffffff” text_line_height=”1.2em” use_border_color=”off” border_style=”solid” disabled=”off”]<\/p>\nThe challenge<\/span><\/h2>\n
The solution<\/span><\/h2>\n
The script<\/span><\/h2>\n
\nfiles while processing<\/span><\/p>\n