Go

For those who are new to any programming language development, they probably start by referencing and learning from the projects of experts or corporate projects. What first comes into view is the various appearances of the Project layout. The Project layout concerns how we organize a Go project. This specifically refers to the arrangement of folders and files. The official Blog titled Organizing a Go module provides suggestions and explanations. Let's read it together!

Before diving into this article, it's helpful to have a basic understanding of Go Module and the use of Go install.

In the article "Organizing a Go module" from the official blog, the content of a Go project is broadly divided into Package and Command, or a combination of both. A Package is what we commonly know as a library, and Command refers to executable Go files that can be compiled and run.

The evolution process is shown in the following diagram

If it's a pure library project intended for others to import into their own Go projects, then the package route can be referred to. For example, "go-linq" is entirely a library project. "go-redis" also falls into this category.

If it's something that needs to be installed onto a machine to become an executable, then the command route can be referred to. An example of this would be "protoc-gen-go".

Basic package

The directory structure shown below has both the go module and the package code in the root directory of the project.

project-root-directory/
  go.mod
  modname.go
  modname_test.go

The modname.go above is the package code, and modname_test.go is the test program related to modname.go.

If you upload this code to a Github repo, such as github.com/someuser/modname, then the path specified in go.mod for this module would also be github.com/someuser/modname. The details about this will be covered later when we discuss the intricacies of the go module.

module github.com/someuser/modname

Because the package code is in the root directory, it's expected that the initial package name of this package code would be modname.

package modname

Others can then acquire this public package through go get github.com/someuser/modname.

The code of this package can also be split into multiple files, as long as all these files are in the same directory level.

project-root-directory/
  go.mod
  modname.go
  modname_test.go
  auth.go
  auth_test.go
  hash.go
  hash_test.go

Besides being in the same directory level, these files also share the same package name, which is modname.

Basic command

If it's an executable program, the most basic project structure would look like this:

project-root-directory/
  go.mod
  auth.go
  auth_test.go
  client.go
  main.go

The main.go file contains the func main. This shouldn't be an issue. Although the filename could also be modname.go, main.go is more of a conventional naming. All the files in this directory will have the package name as main, since in Go, files in the same directory must have the same package name, or the compilation will fail.

Similarly, when the project is uploaded to github.com/someuser/modname,

module github.com/someuser/modname

we can then use go install to download and install it.

go install github.com/someuser/modname@latest

However, typically these executable projects are not that simple. There might be some code that we don't want to be fetched by other projects using go get once published on git. That's when the internal directory comes into play.

Internal Package

Go introduced the internal package mechanism in version 1.4. Besides preventing external users from directly importing it, the internal package is invisible to other modules, acting as a layer of isolation. Thus, any modifications within the internal package won't affect external packages.

It can also only be accessed by packages that are at the same or lower directory levels. This is why most internal packages are placed in the root directory, aiming to allow all packages in the root directory to import the internal package.

project-root-directory/
  go.mod
  main.go
  package1/
    internal/
      interal.go
    package1.go
  package2/
    internal/
      interal.go
    package2.go
  interanl/
    version/
      version.go
    internal.go

Under this definition, only packages that are at the same or lower levels than package1 can import its internal packages, and the same applies to package2. Even main cannot import these two internal packages. However, main can import the internal package located in the root directory, and package1 can do the same.

If one tries to forcefully import it, a compilation error will arise: use of internal package xxxx/internal not allowed.

As for cases where package1 needs to import the root internal package, you need to assign an alias to the imported internal package in order to use it normally, unless referencing a package that's a sub-directory of the internal package. For instance, importing internal/version from the aforementioned directory structure would be fine. If one wishes to specifically import a portion of the internal.go from the internal package, an alias has to be provided.

Package or command with supporting packages

With an understanding of the internal package, we can perceive the internal package as a supporting package. These are the packages we reference internally but don't want them to be directly imported by external projects. For such purposes, the Go official recommendation is to place them in the internal directory.

project-root-directory/
  internal/
    auth/
      auth.go
      auth_test.go
    hash/
      hash.go
      hash_test.go
  go.mod
  modname.go
  modname_test.go

With this goal in mind, we can design modname.go to import the auth package and the hash package. modname.go can be imported in this manner:

import "github.com/someuser/modname/internal/auth"

However, external projects can only import and use the modname package.

Multiple packages

A project can have multiple importable packages, and each package will have its own directory structure for organization.

project-root-directory/
  go.mod
  modname.go
  modname_test.go
  auth/
    auth.go
    auth_test.go
    token/
      token.go
      token_test.go
  hash/
    hash.go
  internal/
    trace/
      trace.go

Such a structure implies that others can import:

github.com/someuser/modname
github.com/someuser/modname/auth
github.com/someuser/modname/hash

They can also import the token package, which is a sub-package of the auth package.

github.com/someuser/modname/auth/token

External projects simply cannot import:

github.com/someuser/modname/internal/trace

In this project structure design, the internal package is deliberately placed in the root directory to allow packages within the project to import it for use. For example, packages inside the project can import and utilize the trace package.

github.com/someuser/modname/internal/trace

Multiple commands

A project can contain multiple executable programs, such as in the examples prog1/main.go and prog2/main.go.

project-root-directory/
  go.mod
  internal/
    ... shared internal packages
  prog1/
    main.go
  prog2/
    main.go

Hence, users can utilize go install to directly install and use them.

$ go install github.com/someuser/modname/prog1@latest
$ go install github.com/someuser/modname/prog2@latest

However, to easily distinguish the directory names of executable commands, it's commonly recommended to place them inside a directory named cmd. This makes it straightforward to identify which ones are importable packages and which ones are executable commands. It's worth noting that these are just recommendations and not mandatory.

Packages and commands in the same repository

If a project provides both importable packages and installable executable commands, the structure usually looks like this:

project-root-directory/
  go.mod
  modname.go
  modname_test.go
  auth/
    auth.go
    auth_test.go
  internal/
    ... internal packages
  cmd/
    prog1/
      main.go
    prog2/
      main.go

Such a structure implies that others can import:

github.com/someuser/modname
github.com/someuser/modname/auth

One can use go install to install cmd/prog1 and cmd/prog2.

$ go install github.com/someuser/modname/cmd/prog1@latest
$ go install github.com/someuser/modname/cmd/prog2@latest

Taking the Prometheus project as an example:

github.com/prometheus/prometheus/
  cmd/
    prometheus/
      main.go
    promtool/
      main.go
  internal/
  model/
  plugins/
  rules/
  scrape/
  scripts/
  storage/
    interface.go

Prometheus offers the prometheus command which can be installed and launched, and it also provides the promtool utility, allowing users to install and perform checks on Prometheus as well as verify various Prometheus configuration files.

Since Prometheus natively only offers local storage functionality, the storage package defines interfaces for other storage services to import and develop. For instance, Mimir by Grafana Lab internally imports many packages from the Prometheus project.

However, you won't find Mimir importing packages from prometheus/internal because those can only be imported within the Prometheus project itself.

Conclusion

The official article mainly focuses on the root directory of the project, the internal directory, and suggests the use of the cmd directory. If our project needs to provide a bunch of packages for other projects to import, some also recommend consolidating them in the pkg directory to keep the root directory tidy. But again, it's just a recommendation.

To me, the directory structure also reflects a boundary in design. How to explore this boundary shouldn't be forced based on this Project Layout guide. The issue in that repo is also discussing that this is not a standard Go project layout.

More importantly, I believe we should return to designing based on scenarios and needs. For instance, the subdomain and bounded context mentioned in DDD (Domain-Driven Design) are based on business-level design layouts. Structuring projects based on such business layouts ensures that project design aligns closely with business objectives, reducing cognitive load and conversion overhead.

Another aspect is about releasing. If some packages within the project are beneficial for other projects, the official recommendation for easier management, release, and version control is to separate them into independent projects. Only packages and commands that have a close life cycle relationship with our system should be centralized within a single project, facilitating both development and deployment.