# Eng Ver. Project Layout in Go For those who are new to any programming language development, they probably start by referencing and learning from the projects of experts or corporate projects. What first comes into view is the various appearances of the **Project layout**. The Project layout concerns how we `organize` a Go project. This specifically refers to the arrangement of folders and files. The official Blog titled [Organizing a Go module](https://go.dev/doc/modules/layout#multiple-packages) provides suggestions and explanations. Let's read it together! Before diving into this article, it's helpful to have a basic understanding of [Go Module](https://go.dev/blog/using-go-modules) and the use of [Go install](https://go.dev/ref/mod#go-install). In the article "[Organizing a Go module](https://go.dev/doc/modules/layout#multiple-packages)" from the official blog, the content of a Go project is broadly divided into Package and Command, or a combination of both. A `Package` is what we commonly know as a library, and `Command` refers to executable Go files that can be compiled and run. The evolution process is shown in the following diagram ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1696857731504/7f86826d-4683-4763-9022-6af144d96b0c.gif align="center") If it's a pure library project intended for others to import into their own Go projects, then the package route can be referred to. For example, "[go-linq](https://github.com/ahmetb/go-linq)" is entirely a library project. "[go-redis](https://github.com/redis/go-redis)" also falls into this category. If it's something that needs to be installed onto a machine to become an executable, then the command route can be referred to. An example of this would be "[protoc-gen-go](https://github.com/protocolbuffers/protobuf-go)". ----- ## Basic package The directory structure shown below has both the go module and the package code in the root directory of the project. ```bash project-root-directory/ go.mod modname.go modname_test.go ``` The `modname.go` above is the package code, and `modname_test.go` is the test program related to modname.go. If you upload this code to a Github repo, such as `github.com/someuser/modname`, then the path specified in go.mod for this module would also be `github.com/someuser/modname`. The details about this will be covered later when we discuss the intricacies of the go module. ```bash module github.com/someuser/modname ``` Because the package code is in the root directory, it's expected that the initial package name of this package code would be `modname`. ```bash package modname ``` Others can then acquire this public package through `go get github.com/someuser/modname`. The code of this package can also be split into multiple files, as long as all these files are in the same directory level. ```bash project-root-directory/ go.mod modname.go modname_test.go auth.go auth_test.go hash.go hash_test.go ``` Besides being in the same directory level, these files also share the same package name, which is modname. ## Basic command If it's an executable program, the most basic project structure would look like this: ```bash project-root-directory/ go.mod auth.go auth_test.go client.go main.go ``` The `main.go` file contains the `func main`. This shouldn't be an issue. Although the filename could also be modname.go, main.go is more of a conventional naming. All the files in this directory will have the package name as `main`, since in Go, files in the same directory must have the same package name, or the compilation will fail. Similarly, when the project is uploaded to `github.com/someuser/modname`, ```arduino module github.com/someuser/modname ``` we can then use `go install` to download and install it. ```ruby go install github.com/someuser/modname@latest ``` However, typically these executable projects are not that simple. There might be some code that we don't want to be fetched by other projects using go get once published on git. That's when the `internal` directory comes into play. ### Internal Package Go introduced the [internal package](https://go.dev/doc/go1.4#internalpackages) mechanism in version 1.4. Besides preventing external users from directly importing it, the internal package is `invisible` to other modules, acting as a layer of isolation. Thus, any modifications within the internal package won't affect external packages. It can also only be accessed by packages that are `at the same or lower directory levels`. This is why most internal packages are placed in the root directory, aiming to allow all packages in the root directory to import the internal package. ```bash project-root-directory/ go.mod main.go package1/ internal/ interal.go package1.go package2/ internal/ interal.go package2.go interanl/ version/ version.go internal.go ``` Under this definition, only packages that are at the same or lower levels than package1 can import its internal packages, and the same applies to package2. Even main cannot import these two internal packages. However, main can import the internal package located in the root directory, and package1 can do the same. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1696858332222/5534c930-2b30-4334-b8ff-a7db44885bbd.png align="center") If one tries to forcefully import it, a compilation error will arise: `use of internal package xxxx/internal not allowed`. As for cases where package1 needs to import the root internal package, you need to assign an alias to the imported internal package in order to use it normally, unless referencing a package that's a sub-directory of the internal package. For instance, importing `internal/version` from the aforementioned directory structure would be fine. If one wishes to specifically import a portion of the internal.go from the internal package, an alias has to be provided. ## Package or command with supporting packages With an understanding of the internal package, we can perceive the internal package as a supporting package. These are the packages we reference internally but don't want them to be directly imported by external projects. For such purposes, the Go official recommendation is to place them in the `internal` directory. ``` project-root-directory/ internal/ auth/ auth.go auth_test.go hash/ hash.go hash_test.go go.mod modname.go modname_test.go ``` With this goal in mind, we can design modname.go to import the auth package and the hash package. modname.go can be imported in this manner: ```go= import "github.com/someuser/modname/internal/auth" ``` However, external projects can only import and use the `modname` package. ## Multiple packages A project can have multiple importable packages, and each package will have its own directory structure for organization. ``` project-root-directory/ go.mod modname.go modname_test.go auth/ auth.go auth_test.go token/ token.go token_test.go hash/ hash.go internal/ trace/ trace.go ``` Such a structure implies that others can import: ``` github.com/someuser/modname github.com/someuser/modname/auth github.com/someuser/modname/hash ``` They can also import the token package, which is a sub-package of the auth package. ``` github.com/someuser/modname/auth/token ``` External projects simply cannot import: ``` github.com/someuser/modname/internal/trace ``` In this project structure design, the internal package is deliberately placed in the root directory to allow packages within the project to import it for use. For example, packages inside the project can import and utilize the trace package. ``` github.com/someuser/modname/internal/trace ``` ## Multiple commands A project can contain multiple executable programs, such as in the examples prog1/main.go and prog2/main.go. ``` project-root-directory/ go.mod internal/ ... shared internal packages prog1/ main.go prog2/ main.go ``` Hence, users can utilize go install to directly install and use them. ```ruby= $ go install github.com/someuser/modname/prog1@latest $ go install github.com/someuser/modname/prog2@latest ``` However, to easily distinguish the directory names of executable commands, it's commonly recommended to place them inside a directory named `cmd`. This makes it straightforward to identify which ones are importable packages and which ones are executable commands. It's worth noting that these are just recommendations and not mandatory. ## Packages and commands in the same repository If a project provides both importable packages and installable executable commands, the structure usually looks like this: ``` project-root-directory/ go.mod modname.go modname_test.go auth/ auth.go auth_test.go internal/ ... internal packages cmd/ prog1/ main.go prog2/ main.go ``` Such a structure implies that others can import: ``` github.com/someuser/modname github.com/someuser/modname/auth ``` One can use go install to install cmd/prog1 and cmd/prog2. ```ruby= $ go install github.com/someuser/modname/cmd/prog1@latest $ go install github.com/someuser/modname/cmd/prog2@latest ``` Taking the [Prometheus](https://github.com/prometheus/prometheus) project as an example: ``` github.com/prometheus/prometheus/ cmd/ prometheus/ main.go promtool/ main.go internal/ model/ plugins/ rules/ scrape/ scripts/ storage/ interface.go ``` Prometheus offers the prometheus command which can be installed and launched, and it also provides the promtool utility, allowing users to install and perform checks on Prometheus as well as verify various Prometheus configuration files. Since Prometheus natively only offers local storage functionality, the storage package defines interfaces for other storage services to import and develop. For instance, [Mimir](https://github.com/grafana/mimir/tree/80ee3c17ee134541904fcefc7e9b066b6288348d) by Grafana Lab internally imports many packages from the Prometheus project. However, you won't find Mimir importing packages from prometheus/internal because those can only be imported within the Prometheus project itself. ## Conclusion The official article mainly focuses on the root directory of the project, the internal directory, and suggests the use of the cmd directory. If our project needs to provide a bunch of packages for other projects to import, some also recommend consolidating them in the `pkg` directory to keep the root directory tidy. But again, it's just a recommendation. To me, the directory structure also reflects a boundary in design. How to explore this boundary shouldn't be forced based on this [Project Layout guide](https://github.com/golang-standards/project-layout). [The issue](https://github.com/golang-standards/project-layout/issues/117) in that repo is also discussing that this is not a standard Go project layout. More importantly, I believe we should return to designing based on scenarios and needs. For instance, the subdomain and bounded context mentioned in DDD (Domain-Driven Design) are based on business-level design layouts. Structuring projects based on such `business layouts` ensures that project design aligns closely with business objectives, reducing cognitive load and conversion overhead. Another aspect is about releasing. If some packages within the project are beneficial for other projects, the official recommendation for easier management, release, and version control is to separate them into independent projects. Only packages and commands that have a close life cycle relationship with our system should be centralized within a single project, facilitating both development and deployment. ## References [Go doc Organizing a Go module](https://go.dev/doc/modules/layout) [Go doc go1.4 Internal packages](https://go.dev/doc/go1.4#internalpackages) [Go 项目目录该怎么组织？官方终于出指南了！](https://tonybai.com/2023/10/05/the-official-guide-of-organizing-go-project/)