{"id":221,"date":"2010-05-21T06:54:34","date_gmt":"2010-05-21T13:54:34","guid":{"rendered":"http:\/\/www.connorgarvey.com\/blog\/?p=221"},"modified":"2010-05-21T06:57:45","modified_gmt":"2010-05-21T13:57:45","slug":"a-threaded-go-language-application-with-semaphores","status":"publish","type":"post","link":"https:\/\/connorgarvey.com\/blog\/?p=221","title":{"rendered":"A threaded Go language application with semaphores"},"content":{"rendered":"

After hearing about Go<\/a> and its love of parallelism<\/a>, I wanted to give it a try. I couldn’t think of any small projects that needed to be threaded until recently.\u00a0 My employer uses an RPM system, but I’m using Ubuntu, which requires deb files.\u00a0 Searching through the RPMs in a browser is a pain, so I wanted a small script that would search for me.\u00a0 Here’s the script with server names changed.<\/p>\n

package main\r\n\r\nimport (\r\n  \"bufio\"\r\n  \"fmt\"\r\n  \"http\"\r\n  \"os\"\r\n  \"regexp\"\r\n  \"strings\"\r\n)\r\n\r\nvar servers = []string {\r\n  \"http:\/\/www.google.com\/search?client=ubuntu&channel=fs&q=go+language&ie=utf-8&oe=utf-8\",\r\n  \"http:\/\/golang.org\/\",\r\n  \"http:\/\/golang.org\/doc\/go_tutorial.html\"}\r\n\r\nvar maxOpenRequests int = 3\r\n\r\nvar rpmRegexp = regexp.MustCompile(\"href=\\\"[^\\\"]+\\\"\")\r\n\r\nfunc main() {\r\n  search := getSearchTerm()\r\n  out := make(chan string)\r\n  done := make(chan int)\r\n  go printer(out)\r\n  for _, url := range servers {\r\n    go searchURL(search, url, out, done)\r\n  }\r\n  for i := 0; i < len(servers); i++ {\r\n    <- done\r\n  }\r\n}\r\n\r\nfunc getSearchTerm() string {\r\n  args := os.Args\r\n  if len(args) != 2 {\r\n    die(\"Please enter one search term\")\r\n  }\r\n  return args[1]\r\n}\r\n\r\nfunc die(message string) {\r\n  fmt.Println(message)\r\n  os.Exit(1)\r\n}\r\n\r\nfunc printer(out chan string) {\r\n  for {\r\n    fmt.Println(<- out)\r\n  }\r\n}\r\n\r\nvar requestSemaphore = make(chan int, maxOpenRequests) \/\/ Integer chanel with a maximum queue size\r\n\r\nfunc searchURL(search string, url string, out chan string, done chan int) {\r\n  requestSemaphore <- 1 \/\/ Block until put in the semaphore queue\r\n  response, realURL, err := http.Get(url)\r\n  if err == nil {\r\n    bufferedReader := bufio.NewReader(response.Body)\r\n    err = searchAll(search, bufferedReader, url, out)\r\n    response.Body.Close()\r\n  }\r\n  if err != nil {\r\n    die(\"Could not read from \" + realURL + \":\" + err.String())\r\n  }\r\n  <- requestSemaphore \/\/ Dequeue from the semaphore\r\n  done <- 1 \/\/ Signal that function is done\r\n}\r\n\r\nfunc searchAll(search string, reader *bufio.Reader, httpRoot string, out chan string) (os.Error) {\r\n  var error os.Error = nil;\r\n  for {\r\n    var line, err = reader.ReadString('\\n')\r\n    if err != nil {\r\n      if err == os.EOF {\r\n        break\r\n      }\r\n      return error\r\n    }\r\n    var hrefs = rpmRegexp.AllMatchesStringIter(line, 0)\r\n    for href := range hrefs {\r\n      start := strings.Index(href, \"\\\"\") + 1\r\n      end := len(href) - 1\r\n      packageFile := href[start : end]\r\n      if strings.Index(packageFile, search) != -1 {\r\n        out <- httpRoot + packageFile\r\n      }\r\n    }\r\n  }\r\n  return error\r\n}\r\n<\/pre>\n
A quick overview<\/h1>\n
Go uses “channels” for communication among threads.<\/p>\n
    \n
  1. The main thread creates two channels, one to receive search output and one to receive “done” responses when a child thread finishes.<\/li>\n
  2. The main thread attaches the output channel to a simple function that prints to standard out.<\/li>\n
  3. The main thread spawns one child thread per server, giving each the output and done channels.<\/li>\n
  4. The main thread loops over the done channel once for each server, waiting until all requests are complete.<\/li>\n
  5. Each child blocks immediately until space is available in a semaphore.<\/li>\n
  6. The child performs the HTTP get, then searches the response for anything matching a regular expression.\u00a0 Any matches are signaled back to the output channel.<\/li>\n
  7. The child opens a space in the semaphore and signals to the main thread that it is done.<\/li>\n<\/ol>\n
    There isn’t much more to say.\u00a0 The Go web site<\/a> has great tutorials<\/a>, so I don’t yet feel the need to write one.<\/p>\n
    By the way, if you’re searching for Go resources, be sure to search for “go language” and not “go”.\u00a0 But then, that’s probably how you found this page \ud83d\ude09<\/p>\n
    package main<\/p>\n
    import (
    \n“bufio”
    \n“fmt”
    \n“http”
    \n“os”
    \n“regexp”
    \n“strings”
    \n)<\/p>\n
    var servers = []string {
    \n“http:\/\/apt.duncllc.com\/dist\/WS4.0\/RedHat\/RPMS\/”,
    \n“http:\/\/apt.duncllc.com\/dist\/WS4.0\/RedHat\/RPMS.extras\/”,
    \n“http:\/\/apt.duncllc.com\/dist\/WS4.0\/RedHat\/RPMS.java\/”}<\/p>\n
    var maxOpenRequests int = 3<\/p>\n
    var rpmRegexp = regexp.MustCompile(“href=\\”[^\\”]+\\””)<\/p>\n
    func main() {
    \nsearch := getSearchTerm()
    \nout := make(chan string)
    \ndone := make(chan int)
    \ngo printer(out)
    \nfor _, url := range servers {
    \ngo searchURL(search, url, out, done)
    \n}
    \nfor i := 0; i < len(servers); i++ {
    \n<- done
    \n}
    \n}<\/p>\n
    func getSearchTerm() string {
    \nargs := os.Args
    \nif len(args) != 2 {
    \ndie(“Please enter one search term”)
    \n}
    \nreturn args[1]
    \n}<\/p>\n
    func die(message string) {
    \nfmt.Println(message)
    \nos.Exit(1)
    \n}<\/p>\n
    func printer(out chan string) {
    \nfor {
    \nfmt.Println(<- out)
    \n}
    \n}<\/p>\n
    var requestSemaphore = make(chan int, maxOpenRequests) \/\/ Integer chanel with a maximum queue size<\/p>\n
    func searchURL(search string, url string, out chan string, done chan int) {
    \nrequestSemaphore <- 1 \/\/ Block until put in the semaphore queue
    \nresponse, realURL, err := http.Get(url)
    \nif err == nil {
    \nbufferedReader := bufio.NewReader(response.Body)
    \nerr = searchAll(search, bufferedReader, url, out)
    \nresponse.Body.Close()
    \n}
    \nif err != nil {
    \ndie(“Could not read from ” + realURL + “:” + err.String())
    \n}
    \n<- requestSemaphore \/\/ Dequeue from the semaphore
    \ndone <- 1 \/\/ Signal that function is done
    \n}<\/p>\n
    func searchAll(search string, reader *bufio.Reader, httpRoot string, out chan string) (os.Error) {
    \nvar error os.Error = nil;
    \nfor {
    \nvar line, err = reader.ReadString(‘\\n’)
    \nif err != nil {
    \nif err == os.EOF {
    \nbreak
    \n}
    \nreturn error
    \n}
    \nvar hrefs = rpmRegexp.AllMatchesStringIter(line, 0)
    \nfor href := range hrefs {
    \nstart := strings.Index(href, “\\””) + 1
    \nend := len(href) – 1
    \npackageFile := href[start : end]
    \nif strings.Index(packageFile, search) != -1 {
    \nout <- httpRoot + packageFile
    \n}
    \n}
    \n}
    \nreturn error
    \n}<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    After hearing about Go and its love of parallelism, I wanted to give it a try. I couldn’t think of any small projects that needed to be threaded until recently.\u00a0 My employer uses an RPM system, but I’m using Ubuntu, which requires deb files.\u00a0 Searching through the RPMs in a browser is a pain, so […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[64],"tags":[42,43,15,5],"class_list":["post-221","post","type-post","status-publish","format-standard","hentry","category-go","tag-go","tag-go-language","tag-open","tag-programming"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1ql2n-3z","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/221","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=221"}],"version-history":[{"count":2,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/221\/revisions"}],"predecessor-version":[{"id":224,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/221\/revisions\/224"}],"wp:attachment":[{"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connorgarvey.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}