/

Using Go to Retrieve a List of GitHub Repositories

Using Go to Retrieve a List of GitHub Repositories

In this blog post, we will explore how to use the Go net/http stdlib package to retrieve information about public repositories from the GitHub API. We will specifically focus on obtaining Go repositories with more than 10k stars.

GitHub provides a convenient public API called Search, which you can find at https://developer.github.com/v3/search/#search-repositories. By making an HTTP GET request to https://api.github.com/search/repositories?q=stars:>=10000+language:go&sort=stars&order=desc, we can retrieve the repositories that match our criteria: Go repositories with more than 10k stars, sorted by the number of stars.

Let’s take a look at a simple snippet of code that uses http.Get from net/http to make the request and ioutil.ReadAll to read the response body:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package main

import (
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	res, err := http.Get("https://api.github.com/search/repositories?q=stars:>=10000+language:go&sort=stars&order=desc")
	if err != nil {
		log.Fatal(err)
	}

	body, err := ioutil.ReadAll(res.Body)
	res.Body.Close()
	if err != nil {
		log.Fatal(err)
	}
	if res.StatusCode != 200 {
		log.Fatal("Unexpected status code", res.StatusCode)
	}

	log.Printf("Body: %s\n", body)
}

The output of this code may not be very readable, but you can use an online JSON formatter to make it easier to understand the structure.

To process the JSON response and generate a table from it, we can use the following code example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"
	"text/tabwriter"
	"time"
)

type Owner struct {
	Login string
}

type Item struct {
	ID              int
	Name            string
	FullName        string `json:"full_name"`
	Owner           Owner
	Description     string
	CreatedAt       string `json:"created_at"`
	StargazersCount int    `json:"stargazers_count"`
}

type JSONData struct {
	Count int    `json:"total_count"`
	Items []Item
}

func main() {
	res, err := http.Get("https://api.github.com/search/repositories?q=stars:>=10000+language:go&sort=stars&order=desc")
	if err != nil {
		log.Fatal(err)
	}
	body, err := ioutil.ReadAll(res.Body)
	res.Body.Close()
	if err != nil {
		log.Fatal(err)
	}
	if res.StatusCode != http.StatusOK {
		log.Fatal("Unexpected status code", res.StatusCode)
	}
	data := JSONData{}
	err = json.Unmarshal(body, &data)
	if err != nil {
		log.Fatal(err)
	}
	printData(data)
}

func printData(data JSONData) {
	log.Printf("Repositories found: %d", data.Count)
	const format = "%v\t%v\t%v\t%v\t\n"
	tw := new(tabwriter.Writer).Init(os.Stdout, 0, 8, 2, ' ', 0)
	fmt.Fprintf(tw, format, "Repository", "Stars", "Created at", "Description")
	fmt.Fprintf(tw, format, "----------", "-----", "----------", "----------")
	for _, i := range data.Items {
		desc := i.Description
		if len(desc) > 50 {
			desc = string(desc[:50]) + "..."
		}
		t, err := time.Parse(time.RFC3339, i.CreatedAt)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Fprintf(tw, format, i.FullName, i.StargazersCount, t.Year(), desc)
	}
	tw.Flush()
}

This code defines three structs to unmarshal the JSON response from GitHub: JSONData, Item, and Owner. We use http.Get to retrieve the response body, ioutil.ReadAll to read it, and json.Unmarshal to unmarshal the JSON into our JSONData struct. The printData function processes the data and generates a nicely formatted table using tabwriter.Writer.

By running this code, you can retrieve and display the repositories along with their star count, creation date, and description in a readable format.

Tags: go, GitHub API, net/http, JSON processing