[](https://listmonk.app)
listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL database as its data store.
[](https://listmonk.app)
Visit [listmonk.app](https://listmonk.app) for more info. Check out the [**live demo**](https://demo.listmonk.app).
## Installation
### Docker
The latest image is available on DockerHub at [`listmonk/listmonk:latest`](https://hub.docker.com/r/listmonk/listmonk/tags?page=1&ordering=last_updated&name=latest).
Download and use the sample [docker-compose.yml](https://github.com/knadh/listmonk/blob/master/docker-compose.yml).
```shell
# Download the compose file to the current directory.
curl -LO https://github.com/knadh/listmonk/raw/master/docker-compose.yml
# Run the services in the background.
docker compose up -d
```
Visit `http://localhost:9000`
See [installation docs](https://listmonk.app/docs/installation)
__________________
### Binary
- Download the [latest release](https://github.com/knadh/listmonk/releases) and extract the listmonk binary.
- `./listmonk --new-config` to generate config.toml. Edit it.
- `./listmonk --install` to setup the Postgres DB (or `--upgrade` to upgrade an existing DB. Upgrades are idempotent and running them multiple times have no side effects).
- Run `./listmonk` and visit `http://localhost:9000`
See [installation docs](https://listmonk.app/docs/installation)
__________________
## Developers
listmonk is free and open source software licensed under AGPLv3. If you are interested in contributing, refer to the [developer setup](https://listmonk.app/docs/developer-setup). The backend is written in Go and the frontend is Vue with Buefy for UI.
## License
listmonk is licensed under the AGPL v3 license.
================================================
FILE: SECURITY.md
================================================
# Reporting security issues
Please refer to https://listmonk.app/docs/security-reports/ first to see the list of non-issues and acceptable-risks before reporting a vulnerability.
================================================
FILE: VERSION
================================================
$Format:%h$
$Format:%D$
================================================
FILE: cmd/admin.go
================================================
package main
import (
"encoding/json"
"fmt"
"net/http"
"syscall"
"time"
"github.com/knadh/listmonk/internal/captcha"
"github.com/labstack/echo/v4"
null "gopkg.in/volatiletech/null.v6"
)
type serverConfig struct {
RootURL string `json:"root_url"`
FromEmail string `json:"from_email"`
PublicSubscription struct {
Enabled bool `json:"enabled"`
CaptchaEnabled bool `json:"captcha_enabled"`
CaptchaProvider null.String `json:"captcha_provider"`
CaptchaKey null.String `json:"captcha_key"`
AltchaComplexity int `json:"altcha_complexity"`
} `json:"public_subscription"`
Privacy struct {
DisableTracking bool `json:"disable_tracking"`
IndividualTracking bool `json:"individual_tracking"`
} `json:"privacy"`
MediaProvider string `json:"media_provider"`
Messengers []string `json:"messengers"`
Langs []i18nLang `json:"langs"`
Lang string `json:"lang"`
Permissions json.RawMessage `json:"permissions"`
Update *AppUpdate `json:"update"`
NeedsRestart bool `json:"needs_restart"`
HasLegacyUser bool `json:"has_legacy_user"`
Version string `json:"version"`
}
// GetServerConfig returns general server config.
func (a *App) GetServerConfig(c echo.Context) error {
out := serverConfig{
RootURL: a.urlCfg.RootURL,
FromEmail: a.cfg.FromEmail,
Lang: a.cfg.Lang,
Permissions: a.cfg.PermissionsRaw,
HasLegacyUser: a.cfg.HasLegacyUser,
Privacy: struct {
DisableTracking bool `json:"disable_tracking"`
IndividualTracking bool `json:"individual_tracking"`
}{
DisableTracking: a.cfg.Privacy.DisableTracking,
IndividualTracking: a.cfg.Privacy.IndividualTracking,
},
}
out.PublicSubscription.Enabled = a.cfg.EnablePublicSubPage
// CAPTCHA.
if a.cfg.Security.Captcha.Altcha.Enabled {
out.PublicSubscription.CaptchaEnabled = true
out.PublicSubscription.CaptchaProvider = null.StringFrom(captcha.ProviderAltcha)
out.PublicSubscription.AltchaComplexity = a.cfg.Security.Captcha.Altcha.Complexity
} else if a.cfg.Security.Captcha.HCaptcha.Enabled {
out.PublicSubscription.CaptchaEnabled = true
out.PublicSubscription.CaptchaProvider = null.StringFrom(captcha.ProviderHCaptcha)
out.PublicSubscription.CaptchaKey = null.StringFrom(a.cfg.Security.Captcha.HCaptcha.Key)
}
out.MediaProvider = a.cfg.MediaUpload.Provider
// Language list.
langList, err := getI18nLangList(a.fs)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
fmt.Sprintf("Error loading language list: %v", err))
}
out.Langs = langList
out.Messengers = make([]string, 0, len(a.messengers))
for _, m := range a.messengers {
out.Messengers = append(out.Messengers, m.Name())
}
a.Lock()
out.NeedsRestart = a.needsRestart
out.Update = a.update
a.Unlock()
out.Version = versionString
return c.JSON(http.StatusOK, okResp{out})
}
// GetDashboardCharts returns chart data points to render ont he dashboard.
func (a *App) GetDashboardCharts(c echo.Context) error {
// Get the chart data from the DB.
out, err := a.core.GetDashboardCharts()
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// GetDashboardCounts returns stats counts to show on the dashboard.
func (a *App) GetDashboardCounts(c echo.Context) error {
// Get the chart data from the DB.
out, err := a.core.GetDashboardCounts()
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// ReloadApp sends a reload signal to the app, causing a full restart.
func (a *App) ReloadApp(c echo.Context) error {
go func() {
<-time.After(time.Millisecond * 500)
// Send the reload signal to trigger the wait loop in main.
a.chReload <- syscall.SIGHUP
}()
return c.JSON(http.StatusOK, okResp{true})
}
================================================
FILE: cmd/archive.go
================================================
package main
import (
"bytes"
"encoding/json"
"html/template"
"net/http"
"net/url"
"github.com/gorilla/feeds"
"github.com/knadh/listmonk/internal/manager"
"github.com/knadh/listmonk/models"
"github.com/labstack/echo/v4"
null "gopkg.in/volatiletech/null.v6"
)
type campArchive struct {
UUID string `json:"uuid"`
Subject string `json:"subject"`
Content string `json:"content"`
CreatedAt null.Time `json:"created_at"`
SendAt null.Time `json:"send_at"`
URL string `json:"url"`
}
// GetCampaignArchives renders the public campaign archives page.
func (a *App) GetCampaignArchives(c echo.Context) error {
// Get archives from the DB.
pg := a.pg.NewFromURL(c.Request().URL.Query())
camps, total, err := a.getCampaignArchives(pg.Offset, pg.Limit, false)
if err != nil {
return err
}
if len(camps) == 0 {
return c.JSON(http.StatusOK, okResp{models.PageResults{
Results: []campArchive{},
}})
}
// Meta.
out := models.PageResults{
Results: camps,
Total: total,
Page: pg.Page,
PerPage: pg.PerPage,
}
return c.JSON(200, okResp{out})
}
// GetCampaignArchivesFeed renders the public campaign archives RSS feed.
func (a *App) GetCampaignArchivesFeed(c echo.Context) error {
var (
pg = a.pg.NewFromURL(c.Request().URL.Query())
showFullContent = a.cfg.EnablePublicArchiveRSSContent
)
// Get archives from the DB.
camps, _, err := a.getCampaignArchives(pg.Offset, pg.Limit, showFullContent)
if err != nil {
return err
}
// Format output for the feed.
out := make([]*feeds.Item, 0, len(camps))
for _, c := range camps {
pubDate := c.CreatedAt.Time
if c.SendAt.Valid {
pubDate = c.SendAt.Time
}
out = append(out, &feeds.Item{
Title: c.Subject,
Link: &feeds.Link{Href: c.URL},
Content: c.Content,
Created: pubDate,
})
}
// Generate the feed.
feed := &feeds.Feed{
Title: a.cfg.SiteName,
Link: &feeds.Link{Href: a.urlCfg.RootURL},
Description: a.i18n.T("public.archiveTitle"),
Items: out,
}
if err := feed.WriteRss(c.Response().Writer); err != nil {
a.log.Printf("error generating archive RSS feed: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.errorProcessingRequest"))
}
return nil
}
// CampaignArchivesPage renders the public campaign archives page.
func (a *App) CampaignArchivesPage(c echo.Context) error {
// Get archives from the DB.
pg := a.pg.NewFromURL(c.Request().URL.Query())
out, total, err := a.getCampaignArchives(pg.Offset, pg.Limit, false)
if err != nil {
return err
}
pg.SetTotal(total)
title := a.i18n.T("public.archiveTitle")
return c.Render(http.StatusOK, "archive", struct {
Title string
Description string
Campaigns []campArchive
TotalPages int
Pagination template.HTML
}{title, title, out, pg.TotalPages, template.HTML(pg.HTML("?page=%d"))})
}
// CampaignArchivePage renders the public campaign archives page.
func (a *App) CampaignArchivePage(c echo.Context) error {
// ID can be the UUID or slug.
var (
idStr = c.Param("id")
uuid, slug string
)
if reUUID.MatchString(idStr) {
uuid = idStr
} else {
slug = idStr
}
// Get the campaign from the DB.
pubCamp, err := a.core.GetArchivedCampaign(0, uuid, slug)
if err != nil || pubCamp.Type != models.CampaignTypeRegular {
notFound := false
// Camppaig doesn't exist.
if er, ok := err.(*echo.HTTPError); ok {
if er.Code == http.StatusBadRequest {
notFound = true
}
} else if pubCamp.Type != models.CampaignTypeRegular {
// Campaign isn't of regular type.
notFound = true
}
// 404.
if notFound {
return c.Render(http.StatusNotFound, tplMessage,
makeMsgTpl(a.i18n.T("public.notFoundTitle"), "", a.i18n.T("public.campaignNotFound")))
}
// Some other internal error.
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign")))
}
// "Compile" the campaign template with appropriate data.
out, err := a.compileArchiveCampaigns([]models.Campaign{pubCamp})
if err != nil {
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign")))
}
// Render the campaign body.
camp := out[0].Campaign
msg, err := a.manager.NewCampaignMessage(camp, out[0].Subscriber)
if err != nil {
a.log.Printf("error rendering campaign: %v", err)
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign")))
}
return c.HTML(http.StatusOK, string(msg.Body()))
}
// CampaignArchivePageLatest renders the latest public campaign.
func (a *App) CampaignArchivePageLatest(c echo.Context) error {
// Get the latest campaign from the DB.
camps, _, err := a.getCampaignArchives(0, 1, true)
if err != nil {
return err
}
if len(camps) == 0 {
return c.Render(http.StatusNotFound, tplMessage,
makeMsgTpl(a.i18n.T("public.notFoundTitle"), "", a.i18n.T("public.campaignNotFound")))
}
camp := camps[0]
return c.HTML(http.StatusOK, camp.Content)
}
// getCampaignArchives fetches the public campaign archives from the DB.
func (a *App) getCampaignArchives(offset, limit int, renderBody bool) ([]campArchive, int, error) {
pubCamps, total, err := a.core.GetArchivedCampaigns(offset, limit)
if err != nil {
return []campArchive{}, total, echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("public.errorFetchingCampaign"))
}
msgs, err := a.compileArchiveCampaigns(pubCamps)
if err != nil {
return []campArchive{}, total, err
}
out := make([]campArchive, 0, len(msgs))
for _, m := range msgs {
camp := m.Campaign
archive := campArchive{
UUID: camp.UUID,
Subject: camp.Subject,
CreatedAt: camp.CreatedAt,
SendAt: camp.SendAt,
}
// The campaign may have a custom slug.
if camp.ArchiveSlug.Valid {
archive.URL, _ = url.JoinPath(a.urlCfg.ArchiveURL, camp.ArchiveSlug.String)
} else {
archive.URL, _ = url.JoinPath(a.urlCfg.ArchiveURL, camp.UUID)
}
// Render the full template body if requested.
if renderBody {
msg, err := a.manager.NewCampaignMessage(camp, m.Subscriber)
if err != nil {
return []campArchive{}, total, err
}
archive.Content = string(msg.Body())
}
out = append(out, archive)
}
return out, total, nil
}
// compileArchiveCampaigns compiles the campaign template with the subscriber data.
func (a *App) compileArchiveCampaigns(camps []models.Campaign) ([]manager.CampaignMessage, error) {
var (
b = bytes.Buffer{}
out = make([]manager.CampaignMessage, 0, len(camps))
)
for _, c := range camps {
camp := c
if err := camp.CompileTemplate(a.manager.TemplateFuncs(&camp)); err != nil {
a.log.Printf("error compiling template: %v", err)
return nil, echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("public.errorFetchingCampaign"))
}
// Load the dummy subscriber meta.
var sub models.Subscriber
if err := json.Unmarshal([]byte(camp.ArchiveMeta), &sub); err != nil {
a.log.Printf("error unmarshalling campaign archive meta: %v", err)
return nil, echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("public.errorFetchingCampaign"))
}
m := manager.CampaignMessage{
Campaign: &camp,
Subscriber: sub,
}
// Render the subject if it's a template.
if camp.SubjectTpl != nil {
if err := camp.SubjectTpl.ExecuteTemplate(&b, models.ContentTpl, m); err != nil {
return nil, err
}
camp.Subject = b.String()
b.Reset()
}
out = append(out, m)
}
return out, nil
}
================================================
FILE: cmd/auth.go
================================================
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"image/png"
"net/http"
"net/mail"
"net/url"
"strings"
"time"
"github.com/knadh/listmonk/internal/auth"
"github.com/knadh/listmonk/internal/i18n"
"github.com/knadh/listmonk/internal/notifs"
"github.com/knadh/listmonk/internal/tmptokens"
"github.com/knadh/listmonk/internal/utils"
"github.com/knadh/listmonk/models"
"github.com/labstack/echo/v4"
"github.com/pquerna/otp/totp"
"github.com/zerodha/simplesessions/v3"
"gopkg.in/volatiletech/null.v6"
)
const (
passwordResetTTL = 30 * time.Minute
twofaTokenTTL = 5 * time.Minute
// Length of reset and 2FA auth tokens.
tmpAuthTokenLen = 64
)
type loginTpl struct {
Title string
Description string
NextURI string
Nonce string
PasswordEnabled bool
OIDCProvider string
OIDCProviderLogo string
Error string
}
type oidcState struct {
Nonce string `json:"nonce"`
Next string `json:"next"`
}
type forgotPasswordTpl struct {
Title string
Description string
Error string
}
type resetPasswordTpl struct {
Title string
Description string
Token string
Email string
Error string
}
type twofaTpl struct {
Title string
Description string
Token string
NextURI string
Error string
}
var (
oidcProviders = map[string]struct{}{
"google.com": {},
"microsoftonline.com": {},
"auth0.com": {},
"github.com": {},
}
)
// LoginPage renders the login page and handles the login form.
func (a *App) LoginPage(c echo.Context) error {
// Has the user been setup?
a.Lock()
needsUserSetup := a.needsUserSetup
a.Unlock()
if needsUserSetup {
return a.LoginSetupPage(c)
}
// Process POST login request.
var loginErr error
if c.Request().Method == http.MethodPost {
loginErr = a.doLogin(c)
if loginErr == nil {
return c.Redirect(http.StatusFound, utils.SanitizeURI(c.FormValue("next")))
}
}
// Render the page, with or without POST.
return a.renderLoginPage(c, loginErr)
}
// LoginSetupPage renders the first time user login page and handles the login form.
func (a *App) LoginSetupPage(c echo.Context) error {
// Process POST login request.
var loginErr error
if c.Request().Method == http.MethodPost {
loginErr = a.doFirstTimeSetup(c)
if loginErr == nil {
a.Lock()
a.needsUserSetup = false
a.Unlock()
return c.Redirect(http.StatusFound, utils.SanitizeURI(c.FormValue("next")))
}
}
// Render the page, with or without POST.
return a.renderLoginSetupPage(c, loginErr)
}
// TwofaPage renders the 2FA verification page and handles the 2FA form submission.
func (a *App) TwofaPage(c echo.Context) error {
var token, next string
if c.Request().Method == http.MethodPost {
token = strings.TrimSpace(c.FormValue("token"))
next = utils.SanitizeURI(c.FormValue("next"))
} else {
token = strings.TrimSpace(c.QueryParam("token"))
next = utils.SanitizeURI(c.QueryParam("next"))
}
// If there's no token, redirect.
if len(token) < tmpAuthTokenLen {
return c.Redirect(http.StatusFound, uriAdmin)
}
if next == "" || next == "/" {
next = uriAdmin
}
// Validate the 2FA temp token.
data, err := tmptokens.Check(token)
if err != nil {
return c.Redirect(http.StatusFound, uriAdmin)
}
userID, ok := data.(int)
if !ok {
return a.renderTwofaPage(c, token, next, a.i18n.T("users.invalidRequest"))
}
// Process the 2FA verification POST request.
if c.Request().Method == http.MethodPost {
return a.doTwofaVerify(c, token, userID, next)
}
// Render the 2FA verification page.
return a.renderTwofaPage(c, token, next, "")
}
// Logout logs a user out.
func (a *App) Logout(c echo.Context) error {
// Delete the session from the DB and cookie.
sess := c.Get(auth.SessionKey).(*simplesessions.Session)
_ = sess.Destroy()
return c.JSON(http.StatusOK, okResp{true})
}
// OIDCLogin initializes an OIDC request and redirects to the OIDC provider for login.
func (a *App) OIDCLogin(c echo.Context) error {
// Verify that the request came from the login page (CSRF).
nonce, err := c.Cookie("nonce")
if err != nil || nonce.Value == "" || nonce.Value != c.FormValue("nonce") {
return echo.NewHTTPError(http.StatusUnauthorized, a.i18n.T("users.invalidRequest"))
}
// Sanitize the URL and make it relative.
next := utils.SanitizeURI(c.FormValue("next"))
if next == "/" {
next = uriAdmin
}
// Preparethe OIDC payload to send to the provider.
state := oidcState{Nonce: nonce.Value, Next: next}
b, err := json.Marshal(state)
if err != nil {
a.log.Printf("error marshalling OIDC state: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Redirect to the external OIDC provider.
return c.Redirect(http.StatusFound, a.auth.GetOIDCAuthURL(base64.URLEncoding.EncodeToString(b), nonce.Value))
}
// OIDCFinish receives the redirect callback from the OIDC provider and completes the handshake.
func (a *App) OIDCFinish(c echo.Context) error {
// Verify that the request actually originated from the login request (which sets the nonce value).
nonce, err := c.Cookie("nonce")
if err != nil || nonce.Value == "" {
return a.renderLoginPage(c, echo.NewHTTPError(http.StatusUnauthorized, a.i18n.T("users.invalidRequest")))
}
// Validate the OIDC token.
oidcToken, claims, err := a.auth.ExchangeOIDCToken(c.Request().URL.Query().Get("code"), nonce.Value)
if err != nil {
return a.renderLoginPage(c, err)
}
// Validate the state.
var state oidcState
stateB, err := base64.URLEncoding.DecodeString(c.QueryParam("state"))
if err != nil {
a.log.Printf("error decoding OIDC state: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
if err := json.Unmarshal(stateB, &state); err != nil {
a.log.Printf("error unmarshalling OIDC state: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
if state.Nonce != nonce.Value {
return a.renderLoginPage(c, echo.NewHTTPError(http.StatusUnauthorized, a.i18n.T("users.invalidRequest")))
}
// Validate e-mail from the claim.
email := strings.TrimSpace(claims.Email)
if email == "" {
return a.renderLoginPage(c, errors.New(a.i18n.Ts("globals.messages.invalidFields", "name", "email")))
}
em, err := mail.ParseAddress(email)
if err != nil {
return a.renderLoginPage(c, err)
}
email = strings.ToLower(em.Address)
claims.Email = email
// Get the user by e-mail received from OIDC.
user, userErr := a.core.GetUser(0, "", email)
if userErr != nil {
// If the user doesn't exist, and auto-creation is enabled, create a new user.
if httpErr, ok := userErr.(*echo.HTTPError); ok && httpErr.Code == http.StatusNotFound && a.cfg.Security.OIDC.AutoCreateUsers {
u, err := a.createOIDCUser(claims, c)
if err != nil {
return a.renderLoginPage(c, err)
}
user = u
userErr = nil
} else {
return a.renderLoginPage(c, userErr)
}
}
// Update the user login state (avatar, logged in date) in the DB.
if err := a.core.UpdateUserLogin(user.ID, claims.Picture); err != nil {
return a.renderLoginPage(c, err)
}
// Set the session in the DB and cookie.
if err := a.auth.SaveSession(user, oidcToken, c); err != nil {
return a.renderLoginPage(c, err)
}
// Redirect to the next page.
return c.Redirect(http.StatusFound, utils.SanitizeURI(state.Next))
}
// ForgotPage renders the forgot password page and handles the forgot password form.
func (a *App) ForgotPage(c echo.Context) error {
// Process the forgot password request.
if c.Request().Method == http.MethodPost {
return a.doForgotPassword(c)
}
// Render the forgot page.
out := forgotPasswordTpl{Title: a.i18n.T("users.forgotPassword")}
return c.Render(http.StatusOK, "admin-forgot-password", out)
}
// ResetPage renders the reset password page and handles the reset password form.
func (a *App) ResetPage(c echo.Context) error {
var (
token = strings.TrimSpace(c.QueryParam("token"))
email = strings.ToLower(strings.TrimSpace(c.QueryParam("email")))
)
// Validate token and email (don't delete it yet, as we may need it for POST).
data, err := tmptokens.Check(email)
if err != nil {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
tk, ok := data.(string)
if !ok || tk != token {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
// Validate that the user exists.
_, err = a.core.GetUser(0, "", email)
if err != nil {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
// Process the reset password request form with the new passwords.
if c.Request().Method == http.MethodPost {
return a.doResetPassword(c, token, email)
}
// Render the reset password form for GET request.
return a.renderResetPasswordPage(c, token, email, "")
}
// renderLoginPage renders the login page and handles the login form.
func (a *App) renderLoginPage(c echo.Context, loginErr error) error {
next := utils.SanitizeURI(c.FormValue("next"))
if next == "/" {
next = uriAdmin
}
var (
oidcProviderName = ""
oidcLogo = ""
)
if a.cfg.Security.OIDC.Enabled {
// Defaults.
oidcProviderName = a.cfg.Security.OIDC.ProviderName
oidcLogo = "oidc.png"
u, err := url.Parse(a.cfg.Security.OIDC.ProviderURL)
if err == nil {
h := strings.Split(u.Hostname(), ".")
// Get the last two h for the root domain
prov := ""
if len(h) >= 2 {
prov = h[len(h)-2] + "." + h[len(h)-1]
} else {
prov = u.Hostname()
}
if oidcProviderName == "" {
oidcProviderName = prov
}
// Lookup the logo in the known providers map.
if _, ok := oidcProviders[prov]; ok {
oidcLogo = prov + ".png"
}
}
}
out := loginTpl{
Title: a.i18n.T("users.login"),
PasswordEnabled: true,
OIDCProvider: oidcProviderName,
OIDCProviderLogo: oidcLogo,
NextURI: next,
}
// If there was an error in the previous state (POST reqest), set it to render in the template.
if loginErr != nil {
if e, ok := loginErr.(*echo.HTTPError); ok {
out.Error = e.Message.(string)
} else {
out.Error = loginErr.Error()
}
}
// Generate and set a nonce for preventing CSRF requests that will be valided in the subsequent requests.
nonce, err := utils.GenerateRandomString(16)
if err != nil {
a.log.Printf("error generating OIDC nonce: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.internalError"))
}
c.SetCookie(&http.Cookie{
Name: "nonce",
Value: nonce,
HttpOnly: true,
Path: "/",
SameSite: http.SameSiteLaxMode,
})
out.Nonce = nonce
// Render the login page.
return c.Render(http.StatusOK, "admin-login", out)
}
// renderLoginSetupPage renders the first time user setup page.
func (a *App) renderLoginSetupPage(c echo.Context, loginErr error) error {
next := utils.SanitizeURI(c.FormValue("next"))
if next == "/" {
next = uriAdmin
}
out := loginTpl{
Title: a.i18n.T("users.login"),
PasswordEnabled: true,
NextURI: next,
}
// If there was an error in the previous state (POST reqest), set it to render in the template.
if loginErr != nil {
if e, ok := loginErr.(*echo.HTTPError); ok {
out.Error = e.Message.(string)
} else {
out.Error = loginErr.Error()
}
}
return c.Render(http.StatusOK, "admin-login-setup", out)
}
// createOIDCUser creates a new user in the DB with the OIDC claims.
func (a *App) createOIDCUser(claims auth.OIDCclaim, c echo.Context) (auth.User, error) {
name := claims.Name
if name == "" {
name = strings.TrimSpace(claims.PreferredUsername)
}
if name == "" {
name = strings.Split(claims.Email, "@")[0]
}
var listRoleID *int
if a.cfg.Security.OIDC.DefaultListRoleID > 0 {
listRoleID = &a.cfg.Security.OIDC.DefaultListRoleID
}
user, err := a.core.CreateUser(auth.User{
Type: auth.UserTypeUser,
HasPassword: false,
PasswordLogin: false,
Username: claims.Email,
Name: name,
Email: null.NewString(claims.Email, true),
UserRoleID: a.cfg.Security.OIDC.DefaultUserRoleID,
ListRoleID: listRoleID,
Status: auth.UserStatusEnabled,
})
return user, err
}
// doLogin logs a user in with a username and password.
func (a *App) doLogin(c echo.Context) error {
var (
startTime = time.Now()
username = strings.TrimSpace(c.FormValue("username"))
password = strings.TrimSpace(c.FormValue("password"))
)
// Ensure timing mitigation is applied regardless of early returns
defer func() {
if elapsed := time.Since(startTime).Milliseconds(); elapsed < 100 {
time.Sleep(time.Duration(100-elapsed) * time.Millisecond)
}
}()
if !strHasLen(username, 3, stdInputMaxLen) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username"))
}
if !strHasLen(password, 8, stdInputMaxLen) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password"))
}
// Log the user in by fetching and verifying credentials from the DB.
user, err := a.core.LoginUser(username, password)
if err != nil {
return err
}
// If TOTP is enabled for the user, create a temp token and redirect to the 2FA page.
if user.TwofaType == models.TwofaTypeTOTP {
// Generate a random token.
token, err := generateRandomString(tmpAuthTokenLen)
if err != nil {
a.log.Printf("error generating 2FA token: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Set the token.
tmptokens.Set(token, twofaTokenTTL, user.ID)
// Redirect to 2FA page.
next := utils.SanitizeURI(c.FormValue("next"))
return c.Redirect(http.StatusFound, fmt.Sprintf("%s/login/twofa?token=%s&next=%s", uriAdmin, token, url.QueryEscape(next)))
}
// Set the session in the DB and cookie.
if err := a.auth.SaveSession(user, "", c); err != nil {
return err
}
return nil
}
// doFirstTimeSetup sets a user up for the first time.
func (a *App) doFirstTimeSetup(c echo.Context) error {
var (
email = strings.TrimSpace(c.FormValue("email"))
username = strings.TrimSpace(c.FormValue("username"))
password = strings.TrimSpace(c.FormValue("password"))
password2 = strings.TrimSpace(c.FormValue("password2"))
)
if !utils.ValidateEmail(email) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "email"))
}
if !strHasLen(username, 3, stdInputMaxLen) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username"))
}
if !strHasLen(password, 8, stdInputMaxLen) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password"))
}
if password != password2 {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("users.passwordMismatch"))
}
// Create the default "Super Admin" with all permissions if it doesn't exist.
if _, err := a.core.GetRole(auth.SuperAdminRoleID); err != nil {
r := auth.Role{
Type: auth.RoleTypeUser,
Name: null.NewString("Super Admin", true),
}
for p := range a.cfg.Permissions {
r.Permissions = append(r.Permissions, p)
}
// Create the role in the DB.
if _, err := a.core.CreateRole(r); err != nil {
return err
}
}
// Create the super admin user in the DB.
u := auth.User{
Type: auth.UserTypeUser,
HasPassword: true,
PasswordLogin: true,
Username: username,
Name: username,
Password: null.NewString(password, true),
Email: null.NewString(email, true),
UserRoleID: auth.SuperAdminRoleID,
Status: auth.UserStatusEnabled,
}
if _, err := a.core.CreateUser(u); err != nil {
return err
}
// Log the user in directly.
user, err := a.core.LoginUser(username, password)
if err != nil {
return err
}
// Set the session in the DB and cookie.
if err := a.auth.SaveSession(user, "", c); err != nil {
return err
}
return nil
}
// renderResetPasswordPage renders the reset password page.
func (a *App) renderResetPasswordPage(c echo.Context, token, email, errMsg string) error {
out := resetPasswordTpl{
Title: a.i18n.T("users.resetPassword"),
Token: token,
Email: email,
Error: errMsg,
}
return c.Render(http.StatusOK, "admin-reset-password", out)
}
// doForgotPassword handles the forgot password form submission.
func (a *App) doForgotPassword(c echo.Context) error {
var (
email = strings.ToLower(strings.TrimSpace(c.FormValue("email")))
)
// Validate email format.
if !utils.ValidateEmail(email) {
return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.resetLinkSent")))
}
// Get the user by email.
user, err := a.core.GetUser(0, "", email)
if err != nil {
return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.resetLinkSent")))
}
// If the password login is disabled, do not proceed, but show success message to prevent email enumeration.
if !user.PasswordLogin {
return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.resetLinkSent")))
}
// Generate a random token.
token, err := generateRandomString(tmpAuthTokenLen)
if err != nil {
a.log.Printf("error generating reset token: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Store the reset token in tmptokens.
tmptokens.Set(email, passwordResetTTL, token)
// Prepare the reset URL.
resetURL := fmt.Sprintf("%s/admin/reset?token=%s&email=%s", a.urlCfg.RootURL, token, url.QueryEscape(email))
// Prepare the email.
var msg bytes.Buffer
data := struct {
ResetURL string
L *i18n.I18n
}{
ResetURL: resetURL,
L: a.i18n,
}
// Render the email template.
if err := notifs.Tpls.ExecuteTemplate(&msg, notifs.TplForgotPassword, data); err != nil {
a.log.Printf("error compiling notification template '%s': %v", notifs.TplForgotPassword, err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
subject, body := notifs.GetTplSubject(a.i18n.T("email.forgotPassword.subject"), msg.Bytes())
// Send the email.
if err := a.emailMsgr.Push(models.Message{
From: a.cfg.FromEmail,
To: []string{email},
Subject: subject,
Body: body,
}); err != nil {
a.log.Printf("error sending reset email: %s", err)
}
// Show the success e-mail nonetheless to prevent e-mail enumeration.
return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.resetLinkSent")))
}
// doResetPassword handles the reset password form submission.
func (a *App) doResetPassword(c echo.Context, token, email string) error {
var (
password = c.FormValue("password")
password2 = c.FormValue("password2")
)
// Validate password.
if !strHasLen(password, 8, stdInputMaxLen) {
return a.renderResetPasswordPage(c, token, email, a.i18n.Ts("globals.messages.invalidFields", "name", "password"))
}
if password != password2 {
return a.renderResetPasswordPage(c, token, email, a.i18n.T("users.passwordMismatch"))
}
// Validate and consume the token (this deletes it).
data, err := tmptokens.Get(email)
if err != nil {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
tk, ok := data.(string)
if !ok || tk != token {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
// Get the user.
user, err := a.core.GetUser(0, "", email)
if err != nil {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("users.invalidResetLink")))
}
// Password login is disabled for the user.
if !user.PasswordLogin {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("users.resetPassword"), "", a.i18n.T("public.invalidFeature")))
}
user.Password = null.NewString(password, true)
if _, err := a.core.UpdateUserProfile(user.ID, user); err != nil {
a.log.Printf("error updating user password: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Log the user in directly without forcing a manual login right after password change.
if err := a.auth.SaveSession(user, "", c); err != nil {
return err
}
// Redirect to the admin page.
return c.Redirect(http.StatusFound, uriAdmin)
}
// renderTwofaPage renders the 2FA verification page.
func (a *App) renderTwofaPage(c echo.Context, token, next, errMsg string) error {
out := twofaTpl{
Title: a.i18n.T("users.twoFA"),
Description: "",
Token: token,
NextURI: next,
Error: errMsg,
}
return c.Render(http.StatusOK, "admin-twofa", out)
}
// doTwofaVerify handles the 2FA verification form submission.
func (a *App) doTwofaVerify(c echo.Context, token string, userID int, next string) error {
totpCode := strings.TrimSpace(c.FormValue("totp_code"))
// Validate.
if !strHasLen(totpCode, 6, 6) {
return a.renderTwofaPage(c, token, next, a.i18n.T("globals.messages.invalidValue"))
}
// Get the user.
user, err := a.core.GetUser(userID, "", "")
if err != nil {
return a.renderTwofaPage(c, token, next, a.i18n.T("users.invalidRequest"))
}
// Verify that TOTP is actually enabled for the user.
if user.TwofaType != models.TwofaTypeTOTP {
return a.renderTwofaPage(c, token, next, a.i18n.T("users.twoFANotEnabled"))
}
// Verify the TOTP code.
valid := totp.Validate(totpCode, user.TwofaKey.String)
if !valid {
return a.renderTwofaPage(c, token, next, a.i18n.T("globals.messages.invalidValue"))
}
// Invalidate the token.
tmptokens.Delete(token)
// Set the session.
if err := a.auth.SaveSession(user, "", c); err != nil {
return err
}
// Redirect to the next page.
return c.Redirect(http.StatusFound, next)
}
// GenerateTOTPQR generates a TOTP QR code for a user to scan with their authenticator app.
func (a *App) GenerateTOTPQR(c echo.Context) error {
u := c.Get(auth.UserHTTPCtxKey).(auth.User)
// If TOTP is already enabled, don't generate a new key.
if u.TwofaType == models.TwofaTypeTOTP {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("users.twoFAAlreadyEnabled"))
}
// Generate a new TOTP key.
key, err := totp.Generate(totp.GenerateOpts{
Issuer: a.cfg.SiteName,
AccountName: u.Email.String,
})
if err != nil {
a.log.Printf("error generating TOTP key: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Convert the TOTP key to a QR code image.
img, err := key.Image(200, 200)
if err != nil {
a.log.Printf("error generating QR code: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
// Encode the QR code as a PNG and return it as base64.
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
a.log.Printf("error encoding QR code: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError"))
}
return c.JSON(http.StatusOK, okResp{struct {
Secret string `json:"secret"`
QR string `json:"qr"`
}{
Secret: key.Secret(),
QR: base64.StdEncoding.EncodeToString(buf.Bytes()),
}})
}
================================================
FILE: cmd/bounce.go
================================================
package main
import (
"encoding/json"
"io"
"net/http"
"strconv"
"time"
"github.com/knadh/listmonk/models"
"github.com/labstack/echo/v4"
)
// GetBounce handles retrieval of a specific bounce record by ID.
func (a *App) GetBounce(c echo.Context) error {
// Fetch one bounce from the DB.
id := getID(c)
out, err := a.core.GetBounce(id)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// GetBounces handles retrieval of bounce records.
func (a *App) GetBounces(c echo.Context) error {
var (
campID, _ = strconv.Atoi(c.QueryParam("campaign_id"))
source = c.FormValue("source")
orderBy = c.FormValue("order_by")
order = c.FormValue("order")
pg = a.pg.NewFromURL(c.Request().URL.Query())
)
// Query and fetch bounces from the DB.
res, total, err := a.core.QueryBounces(campID, 0, source, orderBy, order, pg.Offset, pg.Limit)
if err != nil {
return err
}
// No results.
if len(res) == 0 {
return c.JSON(http.StatusOK, okResp{models.PageResults{Results: []models.Bounce{}}})
}
out := models.PageResults{
Results: res,
Total: total,
Page: pg.Page,
PerPage: pg.PerPage,
}
return c.JSON(http.StatusOK, okResp{out})
}
// GetSubscriberBounces retrieves a subscriber's bounce records.
func (a *App) GetSubscriberBounces(c echo.Context) error {
// Query and fetch bounces from the DB.
subID := getID(c)
out, _, err := a.core.QueryBounces(0, subID, "", "", "", 0, 1000)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// DeleteBounces handles bounce deletion of a list.
func (a *App) DeleteBounces(c echo.Context) error {
all, _ := strconv.ParseBool(c.QueryParam("all"))
var ids []int
if !all {
// There are multiple IDs in the query string.
res, err := parseStringIDs(c.Request().URL.Query()["id"])
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidID", "error", err.Error()))
}
if len(res) == 0 {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidID"))
}
ids = res
}
// Delete bounces from the DB.
if err := a.core.DeleteBounces(ids, all); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{true})
}
// DeleteBounce handles bounce deletion of a single bounce record.
func (a *App) DeleteBounce(c echo.Context) error {
// Delete bounces from the DB.
id := getID(c)
if err := a.core.DeleteBounces([]int{id}, false); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{true})
}
// BlocklistBouncedSubscribers handles blocklisting of all bounced subscribers.
func (a *App) BlocklistBouncedSubscribers(c echo.Context) error {
if err := a.core.BlocklistBouncedSubscribers(); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{true})
}
// BounceWebhook renders the HTML preview of a template.
func (a *App) BounceWebhook(c echo.Context) error {
// Read the request body instead of using c.Bind() to read to save the entire raw request as meta.
rawReq, err := io.ReadAll(c.Request().Body)
if err != nil {
a.log.Printf("error reading ses notification body: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.internalError"))
}
var (
service = c.Param("service")
bounces []models.Bounce
)
switch true {
// Native internal webhook.
case service == "":
var b models.Bounce
if err := json.Unmarshal(rawReq, &b); err != nil {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidData")+":"+err.Error())
}
if bv, err := a.validateBounceFields(b); err != nil {
return err
} else {
b = bv
}
if len(b.Meta) == 0 {
b.Meta = json.RawMessage("{}")
}
if b.CreatedAt.Year() == 0 {
b.CreatedAt = time.Now()
}
bounces = append(bounces, b)
// Amazon SES.
case service == "ses" && a.cfg.BounceSESEnabled:
switch c.Request().Header.Get("X-Amz-Sns-Message-Type") {
// SNS webhook registration confirmation. Only after these are processed will the endpoint
// start getting bounce notifications.
case "SubscriptionConfirmation", "UnsubscribeConfirmation":
if err := a.bounce.SES.ProcessSubscription(rawReq); err != nil {
a.log.Printf("error processing SNS (SES) subscription: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
// Bounce notification.
case "Notification":
b, err := a.bounce.SES.ProcessBounce(rawReq)
if err != nil {
a.log.Printf("error processing SES notification: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
bounces = append(bounces, b)
default:
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
// SendGrid.
case service == "sendgrid" && a.cfg.BounceSendgridEnabled:
var (
sig = c.Request().Header.Get("X-Twilio-Email-Event-Webhook-Signature")
ts = c.Request().Header.Get("X-Twilio-Email-Event-Webhook-Timestamp")
)
// Sendgrid sends multiple bounces.
bs, err := a.bounce.Sendgrid.ProcessBounce(sig, ts, rawReq)
if err != nil {
a.log.Printf("error processing sendgrid notification: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
bounces = append(bounces, bs...)
// Postmark.
case service == "postmark" && a.cfg.BouncePostmarkEnabled:
bs, err := a.bounce.Postmark.ProcessBounce(rawReq, c)
if err != nil {
a.log.Printf("error processing postmark notification: %v", err)
if _, ok := err.(*echo.HTTPError); ok {
return err
}
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
bounces = append(bounces, bs...)
// ForwardEmail.
case service == "forwardemail" && a.cfg.BounceForwardemailEnabled:
var (
sig = c.Request().Header.Get("X-Webhook-Signature")
)
bs, err := a.bounce.Forwardemail.ProcessBounce(sig, rawReq)
if err != nil {
a.log.Printf("error processing forwardemail notification: %v", err)
if _, ok := err.(*echo.HTTPError); ok {
return err
}
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData"))
}
bounces = append(bounces, bs...)
default:
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("bounces.unknownService"))
}
// Insert bounces into the DB.
for _, b := range bounces {
if err := a.bounce.Record(b); err != nil {
a.log.Printf("error recording bounce: %v", err)
}
}
return c.JSON(http.StatusOK, okResp{true})
}
func (a *App) validateBounceFields(b models.Bounce) (models.Bounce, error) {
if b.Email == "" && b.SubscriberUUID == "" {
return b, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "email / subscriber_uuid"))
}
if b.SubscriberUUID != "" && !reUUID.MatchString(b.SubscriberUUID) {
return b, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "subscriber_uuid"))
}
if b.Email != "" {
em, err := a.importer.SanitizeEmail(b.Email)
if err != nil {
return b, echo.NewHTTPError(http.StatusBadRequest, err.Error())
}
b.Email = em
}
if b.Type != models.BounceTypeHard && b.Type != models.BounceTypeSoft && b.Type != models.BounceTypeComplaint {
return b, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "type"))
}
return b, nil
}
================================================
FILE: cmd/campaigns.go
================================================
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"html/template"
"net/http"
"net/url"
"regexp"
"strconv"
"strings"
"time"
"github.com/knadh/listmonk/internal/auth"
"github.com/knadh/listmonk/internal/notifs"
"github.com/knadh/listmonk/models"
"github.com/labstack/echo/v4"
"github.com/lib/pq"
"gopkg.in/volatiletech/null.v6"
)
// campReq is a wrapper over the Campaign model for receiving
// campaign creation and update data from APIs.
type campReq struct {
models.Campaign
// This overrides Campaign.Lists to receive and
// write a list of int IDs during creation and updation.
// Campaign.Lists is JSONText for sending lists children
// to the outside world.
ListIDs []int `json:"lists"`
MediaIDs []int `json:"media"`
// This is only relevant to campaign test requests.
SubscriberEmails pq.StringArray `json:"subscribers"`
}
// campContentReq wraps params coming from API requests for converting
// campaign content formats.
type campContentReq struct {
models.Campaign
From string `json:"from"`
To string `json:"to"`
}
var (
reFromAddress = regexp.MustCompile(`((.+?)\s)?<(.+?)@(.+?)>`)
reSlug = regexp.MustCompile(`[^\p{L}\p{M}\p{N}]`)
)
// GetCampaigns handles retrieval of campaigns.
func (a *App) GetCampaigns(c echo.Context) error {
// Get the authenticated user.
user := auth.GetUser(c)
var (
hasAllPerm = user.HasPerm(auth.PermCampaignsGetAll)
permittedLists []int
)
if !hasAllPerm {
// Either the user has campaigns:get_all permissions and can view all campaigns,
// or the campaigns are filtered by the lists the user has get|manage access to.
hasAllPerm, permittedLists = user.GetPermittedLists(auth.PermTypeGet | auth.PermTypeManage)
}
var (
pg = a.pg.NewFromURL(c.Request().URL.Query())
status = c.QueryParams()["status"]
tags = c.QueryParams()["tag"]
query = strings.TrimSpace(c.FormValue("query"))
orderBy = c.FormValue("order_by")
order = c.FormValue("order")
noBody, _ = strconv.ParseBool(c.QueryParam("no_body"))
)
// Query and retrieve campaigns from the DB.
res, total, err := a.core.QueryCampaigns(query, status, tags, orderBy, order, hasAllPerm, permittedLists, pg.Offset, pg.Limit)
if err != nil {
return err
}
// Remove the body from the response if requested.
if noBody {
for i := range res {
res[i].Body = ""
res[i].BodySource.Valid = false
}
}
// Paginate the response.
if len(res) == 0 {
return c.JSON(http.StatusOK, okResp{models.PageResults{Results: []models.Campaign{}}})
}
out := models.PageResults{
Query: query,
Results: res,
Total: total,
Page: pg.Page,
PerPage: pg.PerPage,
}
return c.JSON(http.StatusOK, okResp{out})
}
// GetCampaign handles retrieval of campaigns.
func (a *App) GetCampaign(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeGet, id, c); err != nil {
return err
}
// Get the campaign from the DB.
out, err := a.core.GetCampaign(id, "", "")
if err != nil {
return err
}
// Blank out the body if requested.
noBody, _ := strconv.ParseBool(c.QueryParam("no_body"))
if noBody {
out.Body = ""
}
return c.JSON(http.StatusOK, okResp{out})
}
// PreviewCampaign renders the HTML preview of a campaign body.
func (a *App) PreviewCampaign(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeGet, id, c); err != nil {
return err
}
var (
isPost = c.Request().Method == http.MethodPost
contentType = c.FormValue("content_type")
tplID, _ = strconv.Atoi(c.FormValue("template_id"))
)
// For visual content, template ID for previewing is irrelevant.
if contentType == models.CampaignContentTypeVisual || tplID < 1 {
tplID = 0
}
// Get the campaign from the DB for previewing with the `template_body` field.
camp, err := a.core.GetCampaignForPreview(id, tplID)
if err != nil {
return err
}
// There's a body in the request to preview instead of the body in the DB.
if isPost {
camp.ContentType = contentType
camp.Body = c.FormValue("body")
// For visual campaigns, template body from the DB shouldn't be used.
if contentType == models.CampaignContentTypeVisual {
camp.TemplateBody = ""
}
}
// Use a dummy campaign ID to prevent views and clicks from {{ TrackView }}
// and {{ TrackLink }} being registered on preview.
camp.UUID = dummySubscriber.UUID
if err := camp.CompileTemplate(a.manager.TemplateFuncs(&camp)); err != nil {
a.log.Printf("error compiling template: %v", err)
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("templates.errorCompiling", "error", err.Error()))
}
// Render the message body.
msg, err := a.manager.NewCampaignMessage(&camp, dummySubscriber)
if err != nil {
a.log.Printf("error rendering message: %v", err)
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("templates.errorRendering", "error", err.Error()))
}
// Plaintext headers for plain body.
if camp.ContentType == models.CampaignContentTypePlain {
return c.String(http.StatusOK, string(msg.Body()))
}
return c.HTML(http.StatusOK, string(msg.Body()))
}
// PreviewCampaignArchive renders the public campaign archives page.
func (a *App) PreviewCampaignArchive(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeGet, id, c); err != nil {
return err
}
// Fetch the campaign body from the DB.
tplID, _ := strconv.Atoi(c.FormValue("template_id"))
camp, err := a.core.GetCampaignForPreview(id, tplID)
if err != nil {
return err
}
camp.ArchiveMeta = json.RawMessage([]byte(c.FormValue("archive_meta")))
// "Compile" the campaign template with appropriate data.
res, err := a.compileArchiveCampaigns([]models.Campaign{camp})
if err != nil {
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign")))
}
// Render the campaign body.
out := res[0].Campaign
msg, err := a.manager.NewCampaignMessage(out, res[0].Subscriber)
if err != nil {
a.log.Printf("error rendering campaign: %v", err)
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign")))
}
return c.HTML(http.StatusOK, string(msg.Body()))
}
// CampaignContent handles campaign content (body) format conversions.
func (a *App) CampaignContent(c echo.Context) error {
var camp campContentReq
if err := c.Bind(&camp); err != nil {
return err
}
// Convert formats, eg: markdown to HTML.
out, err := camp.ConvertContent(camp.From, camp.To)
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest, err.Error())
}
return c.JSON(http.StatusOK, okResp{out})
}
// CreateCampaign handles campaign creation.
// Newly created campaigns are always drafts.
func (a *App) CreateCampaign(c echo.Context) error {
var o campReq
if err := c.Bind(&o); err != nil {
return err
}
// Filter lists against the current user's permitted lists.
user := auth.GetUser(c)
o.ListIDs = user.FilterListsByPerm(auth.PermTypeGet|auth.PermTypeManage, o.ListIDs)
// If the campaign's 'opt-in', prepare a default message.
switch o.Type {
case models.CampaignTypeOptin:
op, err := a.makeOptinCampaignMessage(o)
if err != nil {
return err
}
o = op
case "":
o.Type = models.CampaignTypeRegular
}
if o.Messenger == "" {
o.Messenger = "email"
}
// Validate.
if c, err := a.validateCampaignFields(o); err != nil {
return echo.NewHTTPError(http.StatusBadRequest, err.Error())
} else {
o = c
}
if o.ArchiveTemplateID.Valid && o.ArchiveTemplateID.Int != 0 {
o.ArchiveTemplateID = o.TemplateID
}
out, err := a.core.CreateCampaign(o.Campaign, o.ListIDs, o.MediaIDs)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// UpdateCampaign handles campaign modification.
// Campaigns that are done cannot be modified.
func (a *App) UpdateCampaign(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeManage, id, c); err != nil {
return err
}
// Retrieve the campaign from the DB.
cm, err := a.core.GetCampaign(id, "", "")
if err != nil {
return err
}
if !canEditCampaign(cm.Status) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("campaigns.cantUpdate"))
}
// Clear attribs to avoid merging old and new values as json.Unmarshal in JSON.scan() merges maps,
// merging values already in the DB and incoming values. If this is nil, then DB values remain
// unchanged.
cm.Attribs = nil
// Read the incoming params into the existing campaign fields from the DB.
// This allows updating of values that have been sent whereas fields
// that are not in the request retain the old values.
o := campReq{Campaign: cm}
if err := c.Bind(&o); err != nil {
return err
}
if c, err := a.validateCampaignFields(o); err != nil {
return echo.NewHTTPError(http.StatusBadRequest, err.Error())
} else {
o = c
}
out, err := a.core.UpdateCampaign(id, o.Campaign, o.ListIDs, o.MediaIDs)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// UpdateCampaignStatus handles campaign status modification.
func (a *App) UpdateCampaignStatus(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeManage, id, c); err != nil {
return err
}
req := struct {
Status string `json:"status"`
}{}
if err := c.Bind(&req); err != nil {
return err
}
// Update the campaign status in the DB.
out, err := a.core.UpdateCampaignStatus(id, req.Status)
if err != nil {
return err
}
// If the campaign is being stopped, send the signal to the manager to stop it in flight.
if req.Status == models.CampaignStatusPaused || req.Status == models.CampaignStatusCancelled {
a.manager.StopCampaign(id)
}
return c.JSON(http.StatusOK, okResp{out})
}
// UpdateCampaignArchive handles campaign status modification.
func (a *App) UpdateCampaignArchive(c echo.Context) error {
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeManage, id, c); err != nil {
return err
}
req := struct {
Archive bool `json:"archive"`
TemplateID int `json:"archive_template_id"`
Meta models.JSON `json:"archive_meta"`
ArchiveSlug string `json:"archive_slug"`
}{}
if err := c.Bind(&req); err != nil {
return err
}
if req.ArchiveSlug != "" {
// Format the slug to be alpha-numeric-dash.
s := strings.ToLower(req.ArchiveSlug)
s = strings.TrimSpace(reSlug.ReplaceAllString(s, " "))
s = regexpSpaces.ReplaceAllString(s, "-")
req.ArchiveSlug = s
}
if err := a.core.UpdateCampaignArchive(id, req.Archive, req.TemplateID, req.Meta, req.ArchiveSlug); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{req})
}
// DeleteCampaign handles campaign deletion.
// Only scheduled campaigns that have not started yet can be deleted.
func (a *App) DeleteCampaign(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeManage, id, c); err != nil {
return err
}
// Delete the campaign from the DB.
if err := a.core.DeleteCampaign(id); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{true})
}
// DeleteCampaigns deletes multiple campaigns by IDs or by query.
func (a *App) DeleteCampaigns(c echo.Context) error {
// Get the authenticated user.
user := auth.GetUser(c)
var (
hasAllPerm = user.HasPerm(auth.PermCampaignsManageAll)
permittedLists []int
)
if !hasAllPerm {
// Either the user has campaigns:manage_all permissions and can manage all campaigns,
// or the campaigns are filtered by the lists the user has get|manage access to.
hasAllPerm, permittedLists = user.GetPermittedLists(auth.PermTypeGet | auth.PermTypeManage)
}
var (
ids []int
query string
all bool
)
// Check for IDs in query params.
if len(c.Request().URL.Query()["id"]) > 0 {
var err error
ids, err = parseStringIDs(c.Request().URL.Query()["id"])
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error()))
}
} else {
// Check for query param.
query = strings.TrimSpace(c.FormValue("query"))
all = c.FormValue("all") == "true"
}
// Validate that either IDs or query is provided.
if len(ids) == 0 && (query == "" && !all) {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("globals.messages.errorInvalidIDs", "error", "id or query required"))
}
// Delete the campaigns from the DB.
if err := a.core.DeleteCampaigns(ids, query, hasAllPerm, permittedLists); err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{true})
}
// GetRunningCampaignStats returns stats of a given set of campaign IDs.
func (a *App) GetRunningCampaignStats(c echo.Context) error {
// Get the running campaign stats from the DB.
out, err := a.core.GetRunningCampaignStats()
if err != nil {
return err
}
if len(out) == 0 {
return c.JSON(http.StatusOK, okResp{[]struct{}{}})
}
// Compute rate.
for i, c := range out {
if c.Started.Valid && c.UpdatedAt.Valid {
diff := max(int(c.UpdatedAt.Time.Sub(c.Started.Time).Minutes()), 1)
rate := c.Sent / diff
if rate > c.Sent || rate > c.ToSend {
rate = c.Sent
}
// Rate since the starting of the campaign.
out[i].NetRate = rate
// Realtime running rate over the last minute.
out[i].Rate = a.manager.GetCampaignStats(c.ID).SendRate
}
}
return c.JSON(http.StatusOK, okResp{out})
}
// TestCampaign handles the sending of a campaign message to
// arbitrary subscribers for testing.
func (a *App) TestCampaign(c echo.Context) error {
// Get the campaign ID.
id := getID(c)
// Check if the user has access to the campaign.
if err := a.checkCampaignPerm(auth.PermTypeManage, id, c); err != nil {
return err
}
// Get and validate fields.
var req campReq
if err := c.Bind(&req); err != nil {
return err
}
// Validate.
if c, err := a.validateCampaignFields(req); err != nil {
return echo.NewHTTPError(http.StatusBadRequest, err.Error())
} else {
req = c
}
if len(req.SubscriberEmails) == 0 {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("campaigns.noSubsToTest"))
}
// Sanitize subscriber e-mails.
for i := range req.SubscriberEmails {
req.SubscriberEmails[i] = strings.ToLower(strings.TrimSpace(req.SubscriberEmails[i]))
}
// Get the subscribers from the DB by their e-mails.
subs, err := a.core.GetSubscribersByEmail(req.SubscriberEmails)
if err != nil {
return err
}
// Get the campaign from the DB for previewing.
tplID, _ := strconv.Atoi(c.FormValue("template_id"))
camp, err := a.core.GetCampaignForPreview(id, tplID)
if err != nil {
return err
}
// Override certain values from the DB with incoming values.
camp.Name = req.Name
camp.Subject = req.Subject
camp.FromEmail = req.FromEmail
camp.Body = req.Body
camp.AltBody = req.AltBody
camp.Messenger = req.Messenger
camp.ContentType = req.ContentType
camp.Headers = req.Headers
camp.TemplateID = req.TemplateID
for _, id := range req.MediaIDs {
if id > 0 {
camp.MediaIDs = append(camp.MediaIDs, int64(id))
}
}
// Send the test messages.
for _, s := range subs {
sub := s
if err := a.sendTestMessage(sub, &camp); err != nil {
a.log.Printf("error sending test message: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("campaigns.errorSendTest", "error", err.Error()))
}
}
return c.JSON(http.StatusOK, okResp{true})
}
// GetCampaignViewAnalytics retrieves view counts for a campaign.
func (a *App) GetCampaignViewAnalytics(c echo.Context) error {
ids, err := parseStringIDs(c.Request().URL.Query()["id"])
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error()))
}
if len(ids) == 0 {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("globals.messages.missingFields", "name", "`id`"))
}
var (
typ = c.Param("type")
from = c.QueryParams().Get("from")
to = c.QueryParams().Get("to")
)
if !strHasLen(from, 10, 30) || !strHasLen(to, 10, 30) {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("analytics.invalidDates"))
}
// Campaign link stats.
if typ == "links" {
out, err := a.core.GetCampaignAnalyticsLinks(ids, typ, from, to)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// Get the analytics numbers from the DB for the campaigns.
out, err := a.core.GetCampaignAnalyticsCounts(ids, typ, from, to)
if err != nil {
return err
}
return c.JSON(http.StatusOK, okResp{out})
}
// sendTestMessage takes a campaign and a subscriber and sends out a sample campaign message.
func (a *App) sendTestMessage(sub models.Subscriber, camp *models.Campaign) error {
if err := camp.CompileTemplate(a.manager.TemplateFuncs(camp)); err != nil {
a.log.Printf("error compiling template: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("templates.errorCompiling", "error", err.Error()))
}
// Create a sample campaign message.
msg, err := a.manager.NewCampaignMessage(camp, sub)
if err != nil {
a.log.Printf("error rendering message: %v", err)
return echo.NewHTTPError(http.StatusNotFound, a.i18n.Ts("templates.errorRendering", "error", err.Error()))
}
return a.manager.PushCampaignMessage(msg)
}
// validateCampaignFields validates incoming campaign field values.
func (a *App) validateCampaignFields(c campReq) (campReq, error) {
if c.FromEmail == "" {
c.FromEmail = a.cfg.FromEmail
} else if !reFromAddress.Match([]byte(c.FromEmail)) {
if _, err := a.importer.SanitizeEmail(c.FromEmail); err != nil {
return c, errors.New(a.i18n.T("campaigns.fieldInvalidFromEmail"))
}
}
if !strHasLen(c.Name, 1, stdInputMaxLen) {
return c, errors.New(a.i18n.T("campaigns.fieldInvalidName"))
}
// Larger char limit for subject as it can contain {{ go templating }} logic.
if !strHasLen(c.Subject, 1, 5000) {
return c, errors.New(a.i18n.T("campaigns.fieldInvalidSubject"))
}
// If no content-type is specified, default to richtext.
if c.ContentType != models.CampaignContentTypeRichtext &&
c.ContentType != models.CampaignContentTypeHTML &&
c.ContentType != models.CampaignContentTypePlain &&
c.ContentType != models.CampaignContentTypeVisual &&
c.ContentType != models.CampaignContentTypeMarkdown {
c.ContentType = models.CampaignContentTypeRichtext
}
if c.ContentType != models.CampaignContentTypeVisual {
c.BodySource.Valid = false
}
// If there's a "send_at" date, it should be in the future.
if c.SendAt.Valid {
if c.SendAt.Time.Before(time.Now()) {
return c, errors.New(a.i18n.T("campaigns.fieldInvalidSendAt"))
}
}
if len(c.ListIDs) == 0 {
return c, errors.New(a.i18n.T("campaigns.fieldInvalidListIDs"))
}
if !a.manager.HasMessenger(c.Messenger) {
// If it's a specific SMTP, but it's no longer available (removed/disabled), fall back to general email messenger.
if strings.HasPrefix(c.Messenger, "email-") {
c.Messenger = "email"
} else {
return c, errors.New(a.i18n.Ts("campaigns.fieldInvalidMessenger", "name", c.Messenger))
}
}
camp := models.Campaign{Body: c.Body, TemplateBody: tplTag}
if err := c.CompileTemplate(a.manager.TemplateFuncs(&camp)); err != nil {
return c, errors.New(a.i18n.Ts("campaigns.fieldInvalidBody", "error", err.Error()))
}
if len(c.Headers) == 0 {
c.Headers = make([]map[string]string, 0)
}
// Validate and initialize attribs.
if c.Attribs != nil {
if _, err := json.Marshal(c.Attribs); err != nil {
return c, errors.New(a.i18n.T("subscribers.invalidJSON"))
}
}
if len(c.ArchiveMeta) == 0 {
c.ArchiveMeta = json.RawMessage("{}")
}
if c.ArchiveSlug.String != "" {
// Format the slug to be alpha-numeric-dash.
s := strings.ToLower(c.ArchiveSlug.String)
s = strings.TrimSpace(reSlug.ReplaceAllString(s, " "))
s = regexpSpaces.ReplaceAllString(s, "-")
c.ArchiveSlug = null.NewString(s, true)
} else {
// If there's no slug set, set it to NULL in the DB.
c.ArchiveSlug.Valid = false
}
return c, nil
}
// makeOptinCampaignMessage makes a default opt-in campaign message body.
func (a *App) makeOptinCampaignMessage(o campReq) (campReq, error) {
if len(o.ListIDs) == 0 {
return o, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("campaigns.fieldInvalidListIDs"))
}
// Fetch double opt-in lists from the given list IDs from the DB.
lists, err := a.core.GetListsByOptin(o.ListIDs, models.ListOptinDouble)
if err != nil {
return o, err
}
// There are no double opt-in lists.
if len(lists) == 0 {
return o, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("campaigns.noOptinLists"))
}
// Construct the opt-in URL with list IDs.
listIDs := url.Values{}
for _, l := range lists {
listIDs.Add("l", l.UUID)
}
// optinURLFunc := template.URL("{{ OptinURL }}?" + listIDs.Encode())
optinURLAttr := template.HTMLAttr(fmt.Sprintf(`href="{{ OptinURL }}%s"`, listIDs.Encode()))
// Prepare sample opt-in message for the campaign.
var b bytes.Buffer
if err := notifs.Tpls.ExecuteTemplate(&b, "optin-campaign", struct {
Lists []models.List
OptinURLAttr template.HTMLAttr
}{lists, optinURLAttr}); err != nil {
a.log.Printf("error compiling 'optin-campaign' template: %v", err)
return o, echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("templates.errorCompiling", "error", err.Error()))
}
o.Body = b.String()
return o, nil
}
// checkCampaignPerm checks if the user has get or manage access to the given campaign.
// Either the user has blanket get_all/manage_all permissions, or the campaign
// belongs to lists that the user has access to.
func (a *App) checkCampaignPerm(types auth.PermType, id int, c echo.Context) error {
// Get the authenticated user.
user := auth.GetUser(c)
perm := auth.PermCampaignsGet
if types&auth.PermTypeGet != 0 {
// It's a get request and there's a blanket get all permission.
if user.HasPerm(auth.PermCampaignsGetAll) {
return nil
}
} else {
// It's a manage request and there's a blanket manage_all permission.
if user.HasPerm(auth.PermCampaignsManageAll) {
return nil
}
perm = auth.PermCampaignsManage
}
// There are no *_all campaign permissions. Instead, check if the user access
// blanket get_all/manage_all list permissions. If yes, then the user can access
// all campaigns. If there are no *_all permissions, then ensure that the
// campaign belongs to the lists that the user has access to.
if hasAllPerm, permittedListIDs := user.GetPermittedLists(auth.PermTypeGet | auth.PermTypeManage); !hasAllPerm {
if ok, err := a.core.CampaignHasLists(id, permittedListIDs); err != nil {
return err
} else if !ok {
return echo.NewHTTPError(http.StatusForbidden,
a.i18n.Ts("globals.messages.permissionDenied", "name", perm))
}
}
return nil
}
// canEditCampaign returns true if a campaign is in a status where updating
// its properties is allowed.
func canEditCampaign(status string) bool {
return status == models.CampaignStatusDraft ||
status == models.CampaignStatusPaused ||
status == models.CampaignStatusScheduled
}
================================================
FILE: cmd/events.go
================================================
package main
import (
"encoding/json"
"fmt"
"log"
"time"
"github.com/labstack/echo/v4"
)
// EventStream serves an endpoint that never closes and pushes a
// live event stream (text/event-stream) such as a error messages.
func (a *App) EventStream(c echo.Context) error {
hdr := c.Response().Header()
hdr.Set(echo.HeaderContentType, "text/event-stream")
hdr.Set(echo.HeaderCacheControl, "no-store")
hdr.Set(echo.HeaderConnection, "keep-alive")
// Subscribe to the event stream with a random ID.
id := fmt.Sprintf("api:%v", time.Now().UnixNano())
sub, err := a.events.Subscribe(id)
if err != nil {
log.Fatalf("error subscribing to events: %v", err)
}
ctx := c.Request().Context()
for {
select {
case e := <-sub:
b, err := json.Marshal(e)
if err != nil {
a.log.Printf("error marshalling event: %v", err)
continue
}
c.Response().Write([]byte(fmt.Sprintf("retry: 3000\ndata: %s\n\n", b)))
c.Response().Flush()
case <-ctx.Done():
// On HTTP connection close, unsubscribe.
a.events.Unsubscribe(id)
return nil
}
}
}
================================================
FILE: cmd/handlers.go
================================================
package main
import (
"bytes"
"net/http"
"net/url"
"path"
"regexp"
"strconv"
"github.com/knadh/listmonk/internal/auth"
"github.com/labstack/echo/v4"
"github.com/labstack/echo/v4/middleware"
)
const (
// stdInputMaxLen is the maximum allowed length for a standard input field.
stdInputMaxLen = 2000
// URIs.
uriAdmin = "/admin"
)
type okResp struct {
Data any `json:"data"`
}
var (
reUUID = regexp.MustCompile("^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$")
)
// registerHandlers registers HTTP handlers.
func initHTTPHandlers(e *echo.Echo, a *App) {
// Default error handler.
e.HTTPErrorHandler = func(err error, c echo.Context) {
// Generic, non-echo error. Log it.
if _, ok := err.(*echo.HTTPError); !ok {
a.log.Println(err.Error())
}
e.DefaultHTTPErrorHandler(err, c)
}
// Configure CORS middleware if domains are configured.
if len(a.cfg.Security.CorsOrigins) > 0 {
e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
AllowOrigins: a.cfg.Security.CorsOrigins,
AllowHeaders: []string{echo.HeaderOrigin, echo.HeaderContentType, echo.HeaderAccept},
}))
}
// =================================================================
// Authenticated non /api handlers.
{
// Attach a middleware to the group that checks for auth.
g := e.Group("", a.auth.Middleware, func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
u := c.Get(auth.UserHTTPCtxKey)
// On no-auth, redirect to login page
if _, ok := u.(*echo.HTTPError); ok {
u, _ := url.Parse(a.urlCfg.LoginURL)
q := url.Values{}
q.Set("next", c.Request().RequestURI)
u.RawQuery = q.Encode()
return c.Redirect(http.StatusTemporaryRedirect, u.String())
}
return next(c)
}
})
// Authenticated endpoints.
g.GET(path.Join(uriAdmin, ""), a.AdminPage)
g.GET(path.Join(uriAdmin, "/custom.css"), serveCustomAppearance("admin.custom_css"))
g.GET(path.Join(uriAdmin, "/custom.js"), serveCustomAppearance("admin.custom_js"))
g.GET(path.Join(uriAdmin, "/*"), a.AdminPage)
}
// =================================================================
// Authenticated /api/* handlers.
{
var (
// Permission check middleware.
pm = a.auth.Perm
// Attach a middleware to the group that checks for auth.
g = e.Group("", a.auth.Middleware, func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
u := c.Get(auth.UserHTTPCtxKey)
// On no-auth, respond with a JSON error.
if err, ok := u.(*echo.HTTPError); ok {
return err
}
return next(c)
}
})
)
// API endpoints.
g.GET("/api/health", a.HealthCheck)
g.GET("/api/config", a.GetServerConfig)
g.GET("/api/lang/:lang", a.GetI18nLang)
g.GET("/api/dashboard/charts", a.GetDashboardCharts)
g.GET("/api/dashboard/counts", a.GetDashboardCounts)
g.GET("/api/settings", pm(a.GetSettings, "settings:get"))
g.PUT("/api/settings", pm(a.UpdateSettings, "settings:manage"))
g.PUT("/api/settings/:key", pm(a.UpdateSettingsByKey, "settings:manage"))
g.POST("/api/settings/smtp/test", pm(a.TestSMTPSettings, "settings:manage"))
g.POST("/api/admin/reload", pm(a.ReloadApp, "settings:manage"))
g.GET("/api/logs", pm(a.GetLogs, "settings:get"))
g.GET("/api/events", pm(a.EventStream, "settings:get"))
g.GET("/api/about", a.GetAboutInfo)
g.GET("/api/subscribers", pm(a.QuerySubscribers, "subscribers:get_all", "subscribers:get"))
g.GET("/api/subscribers/:id", pm(hasID(a.GetSubscriber), "subscribers:get_all", "subscribers:get"))
g.GET("/api/subscribers/:id/activity", pm(hasID(a.GetSubscriberActivity), "subscribers:get_all", "subscribers:get"))
g.GET("/api/subscribers/:id/export", pm(hasID(a.ExportSubscriberData), "subscribers:get_all", "subscribers:get"))
g.GET("/api/subscribers/:id/bounces", pm(hasID(a.GetSubscriberBounces), "bounces:get"))
g.DELETE("/api/subscribers/:id/bounces", pm(hasID(a.DeleteSubscriberBounces), "bounces:manage"))
g.POST("/api/subscribers", pm(a.CreateSubscriber, "subscribers:manage"))
g.PUT("/api/subscribers/:id", pm(hasID(a.UpdateSubscriber), "subscribers:manage"))
g.POST("/api/subscribers/:id/optin", pm(hasID(a.SubscriberSendOptin), "subscribers:manage"))
g.PUT("/api/subscribers/blocklist", pm(a.BlocklistSubscribers, "subscribers:manage"))
g.PUT("/api/subscribers/:id/blocklist", pm(hasID(a.BlocklistSubscriber), "subscribers:manage"))
g.PUT("/api/subscribers/lists/:id", pm(a.ManageSubscriberLists, "subscribers:manage"))
g.PUT("/api/subscribers/lists", pm(a.ManageSubscriberLists, "subscribers:manage"))
g.DELETE("/api/subscribers/:id", pm(hasID(a.DeleteSubscriber), "subscribers:manage"))
g.DELETE("/api/subscribers", pm(a.DeleteSubscribers, "subscribers:manage"))
g.GET("/api/bounces", pm(a.GetBounces, "bounces:get"))
g.PUT("/api/bounces/blocklist", pm(a.BlocklistBouncedSubscribers, "bounces:manage"))
g.GET("/api/bounces/:id", pm(hasID(a.GetBounce), "bounces:get"))
g.DELETE("/api/bounces", pm(a.DeleteBounces, "bounces:manage"))
g.DELETE("/api/bounces/:id", pm(hasID(a.DeleteBounce), "bounces:manage"))
// Subscriber operations based on arbitrary SQL queries.
// These aren't very REST-like.
g.POST("/api/subscribers/query/delete", pm(a.DeleteSubscribersByQuery, "subscribers:manage"))
g.PUT("/api/subscribers/query/blocklist", pm(a.BlocklistSubscribersByQuery, "subscribers:manage"))
g.PUT("/api/subscribers/query/lists", pm(a.ManageSubscriberListsByQuery, "subscribers:manage"))
g.GET("/api/subscribers/export",
pm(middleware.GzipWithConfig(middleware.GzipConfig{Level: 9})(a.ExportSubscribers), "subscribers:get_all", "subscribers:get"))
g.GET("/api/import/subscribers", pm(a.GetImportSubscribers, "subscribers:import"))
g.GET("/api/import/subscribers/logs", pm(a.GetImportSubscriberStats, "subscribers:import"))
g.POST("/api/import/subscribers", pm(a.ImportSubscribers, "subscribers:import"))
g.DELETE("/api/import/subscribers", pm(a.StopImportSubscribers, "subscribers:import"))
// Individual list permissions are applied directly within handleGetLists.
g.GET("/api/lists", a.GetLists)
g.GET("/api/lists/:id", hasID(a.GetList))
g.POST("/api/lists", pm(a.CreateList, "lists:manage_all"))
g.PUT("/api/lists/:id", hasID(a.UpdateList))
g.DELETE("/api/lists", a.DeleteLists)
g.DELETE("/api/lists/:id", hasID(a.DeleteList))
g.GET("/api/campaigns", pm(a.GetCampaigns, "campaigns:get_all", "campaigns:get"))
g.GET("/api/campaigns/running/stats", pm(a.GetRunningCampaignStats, "campaigns:get_all", "campaigns:get"))
g.GET("/api/campaigns/:id", pm(hasID(a.GetCampaign), "campaigns:get_all", "campaigns:get"))
g.GET("/api/campaigns/analytics/:type", pm(a.GetCampaignViewAnalytics, "campaigns:get_analytics"))
g.GET("/api/campaigns/:id/preview", pm(hasID(a.PreviewCampaign), "campaigns:get_all", "campaigns:get"))
g.POST("/api/campaigns/:id/preview/archive", pm(hasID(a.PreviewCampaignArchive), "campaigns:get_all", "campaigns:get"))
g.POST("/api/campaigns/:id/preview", pm(hasID(a.PreviewCampaign), "campaigns:get_all", "campaigns:get"))
g.POST("/api/campaigns/:id/content", pm(hasID(a.CampaignContent), "campaigns:manage_all", "campaigns:manage"))
g.POST("/api/campaigns/:id/text", pm(hasID(a.PreviewCampaign), "campaigns:get"))
g.POST("/api/campaigns/:id/test", pm(hasID(a.TestCampaign), "campaigns:manage_all", "campaigns:manage"))
g.POST("/api/campaigns", pm(a.CreateCampaign, "campaigns:manage_all", "campaigns:manage"))
g.PUT("/api/campaigns/:id", pm(hasID(a.UpdateCampaign), "campaigns:manage_all", "campaigns:manage"))
g.PUT("/api/campaigns/:id/status", pm(hasID(a.UpdateCampaignStatus), "campaigns:manage_all", "campaigns:manage"))
g.PUT("/api/campaigns/:id/archive", pm(hasID(a.UpdateCampaignArchive), "campaigns:manage_all", "campaigns:manage"))
g.DELETE("/api/campaigns", pm(a.DeleteCampaigns, "campaigns:manage", "campaigns:manage_all"))
g.DELETE("/api/campaigns/:id", pm(hasID(a.DeleteCampaign), "campaigns:manage_all", "campaigns:manage"))
g.GET("/api/media", pm(a.GetAllMedia, "media:get"))
g.GET("/api/media/:id", pm(hasID(a.GetMedia), "media:get"))
g.POST("/api/media", pm(a.UploadMedia, "media:manage"))
g.DELETE("/api/media/:id", pm(hasID(a.DeleteMedia), "media:manage"))
g.GET("/api/templates", pm(a.GetTemplates, "templates:get"))
g.GET("/api/templates/:id", pm(hasID(a.GetTemplate), "templates:get"))
g.GET("/api/templates/:id/preview", pm(hasID(a.PreviewTemplate), "templates:get"))
g.POST("/api/templates/preview", pm(a.PreviewTemplateBody, "templates:get"))
g.POST("/api/templates", pm(a.CreateTemplate, "templates:manage"))
g.PUT("/api/templates/:id", pm(hasID(a.UpdateTemplate), "templates:manage"))
g.PUT("/api/templates/:id/default", pm(hasID(a.TemplateSetDefault), "templates:manage"))
g.DELETE("/api/templates/:id", pm(hasID(a.DeleteTemplate), "templates:manage"))
g.DELETE("/api/maintenance/subscribers/:type", pm(a.GCSubscribers, "settings:maintain"))
g.DELETE("/api/maintenance/analytics/:type", pm(a.GCCampaignAnalytics, "settings:maintain"))
g.DELETE("/api/maintenance/subscriptions/unconfirmed", pm(a.GCSubscriptions, "settings:maintain"))
g.POST("/api/tx", pm(a.SendTxMessage, "tx:send"))
g.GET("/api/profile", a.GetUserProfile)
g.PUT("/api/profile", a.UpdateUserProfile)
g.GET("/api/users", pm(a.GetUsers, "users:get"))
g.GET("/api/users/:id", pm(hasID(a.GetUser), "users:get"))
g.POST("/api/users", pm(a.CreateUser, "users:manage"))
g.PUT("/api/users/:id", pm(hasID(a.UpdateUser), "users:manage"))
g.DELETE("/api/users", pm(a.DeleteUsers, "users:manage"))
g.DELETE("/api/users/:id", pm(hasID(a.DeleteUser), "users:manage"))
g.POST("/api/logout", a.Logout)
// TOTP 2FA endpoints
g.GET("/api/users/:id/twofa/totp", hasID(a.GenerateTOTPQR))
g.PUT("/api/users/:id/twofa", hasID(a.EnableTOTP))
g.DELETE("/api/users/:id/twofa", hasID(a.DisableTOTP))
g.GET("/api/roles/users", pm(a.GetUserRoles, "roles:get"))
g.GET("/api/roles/lists", pm(a.GeListRoles, "roles:get"))
g.POST("/api/roles/users", pm(a.CreateUserRole, "roles:manage"))
g.POST("/api/roles/lists", pm(a.CreateListRole, "roles:manage"))
g.PUT("/api/roles/users/:id", pm(hasID(a.UpdateUserRole), "roles:manage"))
g.PUT("/api/roles/lists/:id", pm(hasID(a.UpdateListRole), "roles:manage"))
g.DELETE("/api/roles/:id", pm(hasID(a.DeleteRole), "roles:manage"))
if a.cfg.BounceWebhooksEnabled {
// Private authenticated bounce endpoint.
g.POST("/webhooks/bounce", pm(a.BounceWebhook, "webhooks:post_bounce"))
}
}
// =================================================================
// Public API endpoints.
{
// Public unauthenticated endpoints.
g := e.Group("")
if a.cfg.BounceWebhooksEnabled {
// Public bounce endpoints for webservices like SES.
g.POST("/webhooks/service/:service", a.BounceWebhook)
}
// Landing page.
g.GET("/", func(c echo.Context) error {
return c.Render(http.StatusOK, "home", publicTpl{Title: "listmonk"})
})
// Public admin endpoints (login page, OIDC endpoints, password reset).
g.GET(path.Join(uriAdmin, "/login"), a.LoginPage)
g.POST(path.Join(uriAdmin, "/login"), a.LoginPage)
g.GET(path.Join(uriAdmin, "/login/twofa"), a.TwofaPage)
g.POST(path.Join(uriAdmin, "/login/twofa"), a.TwofaPage)
g.GET(path.Join(uriAdmin, "/forgot"), a.ForgotPage)
g.POST(path.Join(uriAdmin, "/forgot"), a.ForgotPage)
g.GET(path.Join(uriAdmin, "/reset"), a.ResetPage)
g.POST(path.Join(uriAdmin, "/reset"), a.ResetPage)
if a.cfg.Security.OIDC.Enabled {
g.POST("/auth/oidc", a.OIDCLogin)
g.GET("/auth/oidc", a.OIDCFinish)
}
// Public APIs.
g.GET("/api/public/lists", a.GetPublicLists)
g.POST("/api/public/subscription", a.PublicSubscription)
g.GET("/api/public/captcha/altcha", a.AltchaChallenge)
if a.cfg.EnablePublicArchive {
g.GET("/api/public/archive", a.GetCampaignArchives)
}
// /public/static/* file server is registered in initHTTPServer().
// Public subscriber facing views.
g.GET("/subscription/form", a.SubscriptionFormPage)
g.POST("/subscription/form", a.SubscriptionForm)
g.GET("/subscription/:campUUID/:subUUID", noIndex(a.hasUUID(a.hasSub(a.SubscriptionPage), "campUUID", "subUUID")))
g.POST("/subscription/:campUUID/:subUUID", a.hasUUID(a.hasSub(a.SubscriptionPrefs), "campUUID", "subUUID"))
g.GET("/subscription/optin/:subUUID", noIndex(a.hasUUID(a.hasSub(a.OptinPage), "subUUID")))
g.POST("/subscription/optin/:subUUID", a.hasUUID(a.hasSub(a.OptinPage), "subUUID"))
g.POST("/subscription/export/:subUUID", a.hasUUID(a.hasSub(a.SelfExportSubscriberData), "subUUID"))
g.POST("/subscription/wipe/:subUUID", a.hasUUID(a.hasSub(a.WipeSubscriberData), "subUUID"))
g.GET("/link/:linkUUID/:campUUID/:subUUID", noIndex(a.hasUUID(a.LinkRedirect, "linkUUID", "campUUID", "subUUID")))
g.GET("/campaign/:campUUID/:subUUID", noIndex(a.hasUUID(a.ViewCampaignMessage, "campUUID", "subUUID")))
g.GET("/campaign/:campUUID/:subUUID/px.png", noIndex(a.hasUUID(a.RegisterCampaignView, "campUUID", "subUUID")))
if a.cfg.EnablePublicArchive {
g.GET("/archive", a.CampaignArchivesPage)
g.GET("/archive.xml", a.GetCampaignArchivesFeed)
g.GET("/archive/:id", a.CampaignArchivePage)
g.GET("/archive/latest", a.CampaignArchivePageLatest)
}
g.GET("/public/custom.css", serveCustomAppearance("public.custom_css"))
g.GET("/public/custom.js", serveCustomAppearance("public.custom_js"))
// Public health API endpoint.
g.GET("/health", a.HealthCheck)
// 404 pages.
g.RouteNotFound("/*", func(c echo.Context) error {
return c.Render(http.StatusNotFound, tplMessage,
makeMsgTpl("404 - "+a.i18n.T("public.notFoundTitle"), "", ""))
})
g.RouteNotFound("/api/*", func(c echo.Context) error {
return echo.NewHTTPError(http.StatusNotFound, "404 unknown endpoint")
})
g.RouteNotFound("/admin/*", func(c echo.Context) error {
return echo.NewHTTPError(http.StatusNotFound, "404 page not found")
})
}
}
// AdminPage is the root handler that renders the Javascript admin frontend.
func (a *App) AdminPage(c echo.Context) error {
b, err := a.fs.Read(path.Join(uriAdmin, "/index.html"))
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, err.Error())
}
b = bytes.ReplaceAll(b, []byte("asset_version"), []byte(a.cfg.AssetVersion))
return c.HTMLBlob(http.StatusOK, b)
}
// HealthCheck is a healthcheck endpoint that returns a 200 response.
func (a *App) HealthCheck(c echo.Context) error {
return c.JSON(http.StatusOK, okResp{true})
}
// serveCustomAppearance serves the given custom CSS/JS appearance blob
// meant for customizing public and admin pages from the admin settings UI.
func serveCustomAppearance(name string) echo.HandlerFunc {
return func(c echo.Context) error {
var (
app = c.Get("app").(*App)
out []byte
hdr string
)
switch name {
case "admin.custom_css":
out = app.cfg.Appearance.AdminCSS
hdr = "text/css; charset=utf-8"
case "admin.custom_js":
out = app.cfg.Appearance.AdminJS
hdr = "application/javascript; charset=utf-8"
case "public.custom_css":
out = app.cfg.Appearance.PublicCSS
hdr = "text/css; charset=utf-8"
case "public.custom_js":
out = app.cfg.Appearance.PublicJS
hdr = "application/javascript; charset=utf-8"
}
return c.Blob(http.StatusOK, hdr, out)
}
}
// hasUUID middleware validates the UUID string format for a given set of params.
func (a *App) hasUUID(next echo.HandlerFunc, params ...string) echo.HandlerFunc {
return func(c echo.Context) error {
for _, p := range params {
if !reUUID.MatchString(c.Param(p)) {
return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "",
a.i18n.T("globals.messages.invalidUUID")))
}
}
return next(c)
}
}
// hasID middleware validates the :id param in the URL and sets its int value in the context.
func hasID(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
id, _ := strconv.Atoi(c.Param("id"))
if id < 1 {
return echo.NewHTTPError(http.StatusBadRequest, "invalid ID")
}
c.Set("id", id)
return next(c)
}
}
// hasSub middleware checks if a subscriber exists given the UUID
// param in a request.
func (a *App) hasSub(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
subUUID := c.Param("subUUID")
if _, err := a.core.GetSubscriber(0, subUUID, ""); err != nil {
if er, ok := err.(*echo.HTTPError); ok && er.Code == http.StatusBadRequest {
return c.Render(http.StatusNotFound, tplMessage,
makeMsgTpl(a.i18n.T("public.notFoundTitle"), "", er.Message.(string)))
}
a.log.Printf("error checking subscriber existence: %v", err)
return c.Render(http.StatusInternalServerError, tplMessage,
makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.errorProcessingRequest")))
}
return next(c)
}
}
// noIndex adds the HTTP header requesting robots to not crawl the page.
func noIndex(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
c.Response().Header().Set("X-Robots-Tag", "noindex")
return next(c)
}
}
// getID returns the :id param from the URL parsed and stored as an int by the hasID middleware.
func getID(c echo.Context) int {
return c.Get("id").(int)
}
================================================
FILE: cmd/i18n.go
================================================
package main
import (
"encoding/json"
"fmt"
"net/http"
"regexp"
"sort"
"github.com/knadh/listmonk/internal/i18n"
"github.com/knadh/stuffbin"
"github.com/labstack/echo/v4"
)
type i18nLang struct {
Code string `json:"code"`
Name string `json:"name"`
}
type i18nLangRaw struct {
Code string `json:"_.code"`
Name string `json:"_.name"`
}
var reLangCode = regexp.MustCompile(`[^a-zA-Z_0-9\\-]`)
// GetI18nLang returns the JSON language pack given the language code.
func (a *App) GetI18nLang(c echo.Context) error {
lang := c.Param("lang")
if len(lang) > 6 || reLangCode.MatchString(lang) {
return echo.NewHTTPError(http.StatusBadRequest, "Invalid language code.")
}
i, ok, err := getI18nLang(lang, a.fs)
if err != nil && !ok {
return echo.NewHTTPError(http.StatusBadRequest, "Unknown language.")
}
return c.JSON(http.StatusOK, okResp{json.RawMessage(i.JSON())})
}
// getI18nLangList returns the list of available i18n languages.
func getI18nLangList(fs stuffbin.FileSystem) ([]i18nLang, error) {
list, err := fs.Glob("/i18n/*.json")
if err != nil {
return nil, err
}
// Read language JSON files from the fs.
var out []i18nLang
for _, l := range list {
b, err := fs.Get(l)
if err != nil {
return out, fmt.Errorf("error reading lang file: %s: %v", l, err)
}
var r i18nLangRaw
if err := json.Unmarshal(b.ReadBytes(), &r); err != nil {
return out, fmt.Errorf("error parsing lang file: %s: %v", l, err)
}
out = append(out, i18nLang(r))
}
// Sort by language code.
sort.SliceStable(out, func(i, j int) bool {
return out[i].Code < out[j].Code
})
return out, nil
}
// The bool indicates whether the specified language could be loaded. If it couldn't
// be, the app shouldn't halt but throw a warning.
func getI18nLang(lang string, fs stuffbin.FileSystem) (*i18n.I18n, bool, error) {
const def = "en"
b, err := fs.Read(fmt.Sprintf("/i18n/%s.json", def))
if err != nil {
return nil, false, fmt.Errorf("error reading default i18n language file: %s: %v", def, err)
}
// Initialize with the default language.
i, err := i18n.New(b)
if err != nil {
return nil, false, fmt.Errorf("error unmarshalling i18n language: %s: %v", lang, err)
}
// Load the selected language on top of it.
b, err = fs.Read(fmt.Sprintf("/i18n/%s.json", lang))
if err != nil {
return i, true, fmt.Errorf("error reading i18n language file: %s: %v", lang, err)
}
if err := i.Load(b); err != nil {
return i, true, fmt.Errorf("error loading i18n language file: %s: %v", lang, err)
}
return i, true, nil
}
================================================
FILE: cmd/import.go
================================================
package main
import (
"encoding/json"
"io"
"net/http"
"os"
"strings"
"github.com/knadh/listmonk/internal/subimporter"
"github.com/knadh/listmonk/models"
"github.com/labstack/echo/v4"
)
// ImportSubscribers handles the uploading and bulk importing of
// a ZIP file of one or more CSV files.
func (a *App) ImportSubscribers(c echo.Context) error {
// Is an import already running?
if a.importer.GetStats().Status == subimporter.StatusImporting {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("import.alreadyRunning"))
}
// Unmarshal the JSON params.
var opt subimporter.SessionOpt
if err := json.Unmarshal([]byte(c.FormValue("params")), &opt); err != nil {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("import.invalidParams", "error", err.Error()))
}
// Validate mode.
if opt.Mode != subimporter.ModeSubscribe && opt.Mode != subimporter.ModeBlocklist {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("import.invalidMode"))
}
// If no status is specified, pick a default one.
if opt.SubStatus == "" {
switch opt.Mode {
case subimporter.ModeSubscribe:
opt.SubStatus = models.SubscriptionStatusUnconfirmed
case subimporter.ModeBlocklist:
opt.SubStatus = models.SubscriptionStatusUnsubscribed
}
}
if opt.SubStatus != models.SubscriptionStatusUnconfirmed &&
opt.SubStatus != models.SubscriptionStatusConfirmed &&
opt.SubStatus != models.SubscriptionStatusUnsubscribed {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("import.invalidSubStatus"))
}
if len(opt.Delim) != 1 {
return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("import.invalidDelim"))
}
// Open the HTTP file.
file, err := c.FormFile("file")
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest,
a.i18n.Ts("import.invalidFile", "error", err.Error()))
}
src, err := file.Open()
if err != nil {
return err
}
defer src.Close()
// Copy it to a temp location.
out, err := os.CreateTemp("", "listmonk")
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("import.errorCopyingFile", "error", err.Error()))
}
defer out.Close()
if _, err = io.Copy(out, src); err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("import.errorCopyingFile", "error", err.Error()))
}
// Start the importer session.
opt.Filename = file.Filename
sess, err := a.importer.NewSession(opt)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("import.errorStarting", "error", err.Error()))
}
go sess.Start()
if strings.HasSuffix(strings.ToLower(file.Filename), ".csv") {
go sess.LoadCSV(out.Name(), rune(opt.Delim[0]))
} else {
// Only 1 CSV from the ZIP is considered. If multiple files have
// to be processed, counting the net number of lines (to track progress),
// keeping the global import state (failed / successful) etc. across
// multiple files becomes complex. Instead, it's just easier for the
// end user to concat multiple CSVs (if there are multiple in the first)
// place and upload as one in the first place.
dir, files, err := sess.ExtractZIP(out.Name(), 1)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
a.i18n.Ts("import.errorProcessingZIP", "error", err.Error()))
}
go sess.LoadCSV(dir+"/"+files[0], rune(opt.Delim[0]))
}
return c.JSON(http.StatusOK, okResp{a.importer.GetStats()})
}
// GetImportSubscribers returns import statistics.
func (a *App) GetImportSubscribers(c echo.Context) error {
s := a.importer.GetStats()
return c.JSON(http.StatusOK, okResp{s})
}
// GetImportSubscriberStats returns import statistics.
func (a *App) GetImportSubscriberStats(c echo.Context) error {
return c.JSON(http.StatusOK, okResp{string(a.importer.GetLogs())})
}
// StopImportSubscribers sends a stop signal to the importer.
// If there's an ongoing import, it'll be stopped, and if an import
// is finished, it's state is cleared.
func (a *App) StopImportSubscribers(c echo.Context) error {
a.importer.Stop()
return c.JSON(http.StatusOK, okResp{a.importer.GetStats()})
}
================================================
FILE: cmd/init.go
================================================
package main
import (
"bytes"
"crypto/md5"
"database/sql"
"encoding/json"
"errors"
"fmt"
"html/template"
"io"
"log"
"log/slog"
"maps"
"net/http"
"os"
"path"
"path/filepath"
"runtime"
"strings"
"syscall"
"time"
"github.com/Masterminds/sprig/v3"
"github.com/gdgvda/cron"
"github.com/jmoiron/sqlx"
"github.com/jmoiron/sqlx/types"
"github.com/knadh/goyesql/v2"
goyesqlx "github.com/knadh/goyesql/v2/sqlx"
koanfmaps "github.com/knadh/koanf/maps"
"github.com/knadh/koanf/parsers/toml"
"github.com/knadh/koanf/providers/confmap"
"github.com/knadh/koanf/providers/file"
"github.com/knadh/koanf/providers/posflag"
"github.com/knadh/koanf/v2"
"github.com/knadh/listmonk/internal/auth"
"github.com/knadh/listmonk/internal/bounce"
"github.com/knadh/listmonk/internal/bounce/mailbox"
"github.com/knadh/listmonk/internal/captcha"
"github.com/knadh/listmonk/internal/core"
"github.com/knadh/listmonk/internal/i18n"
"github.com/knadh/listmonk/internal/manager"
"github.com/knadh/listmonk/internal/media"
"github.com/knadh/listmonk/internal/media/providers/filesystem"
"github.com/knadh/listmonk/internal/media/providers/s3"
"github.com/knadh/listmonk/internal/messenger/email"
"github.com/knadh/listmonk/internal/messenger/postback"
"github.com/knadh/listmonk/internal/notifs"
"github.com/knadh/listmonk/internal/subimporter"
"github.com/knadh/listmonk/models"
"github.com/knadh/stuffbin"
"github.com/labstack/echo/v4"
"github.com/lib/pq"
flag "github.com/spf13/pflag"
"gopkg.in/volatiletech/null.v6"
)
const (
// Path to the SQL queries directory in the embedded FS.
queryFilePath = "/queries"
emailMsgr = "email"
)
// UrlConfig contains various URL constants used in the app.
type UrlConfig struct {
RootURL string `koanf:"root_url"`
LogoURL string `koanf:"logo_url"`
FaviconURL string `koanf:"favicon_url"`
LoginURL string `koanf:"login_url"`
UnsubURL string
LinkTrackURL string
ViewTrackURL string
OptinURL string
MessageURL string
ArchiveURL string
}
// Config contains static, constant config values required by arbitrary handlers and functions.
type Config struct {
SiteName string `koanf:"site_name"`
FromEmail string `koanf:"from_email"`
NotifyEmails []string `koanf:"notify_emails"`
EnablePublicSubPage bool `koanf:"enable_public_subscription_page"`
EnablePublicArchive bool `koanf:"enable_public_archive"`
EnablePublicArchiveRSSContent bool `koanf:"enable_public_archive_rss_content"`
Lang string `koanf:"lang"`
DBBatchSize int `koanf:"batch_size"`
Privacy struct {
IndividualTracking bool `koanf:"individual_tracking"`
DisableTracking bool `koanf:"disable_tracking"`
AllowPreferences bool `koanf:"allow_preferences"`
AllowBlocklist bool `koanf:"allow_blocklist"`
AllowExport bool `koanf:"allow_export"`
AllowWipe bool `koanf:"allow_wipe"`
RecordOptinIP bool `koanf:"record_optin_ip"`
UnsubHeader bool `koanf:"unsubscribe_header"`
Exportable map[string]bool `koanf:"-"`
DomainBlocklist []string `koanf:"-"`
DomainAllowlist []string `koanf:"-"`
} `koanf:"privacy"`
Security struct {
OIDC struct {
Enabled bool `koanf:"enabled"`
ProviderURL string `koanf:"provider_url"`
ProviderName string `koanf:"provider_name"`
ClientID string `koanf:"client_id"`
ClientSecret string `koanf:"client_secret"`
AutoCreateUsers bool `koanf:"auto_create_users"`
DefaultUserRoleID int `koanf:"default_user_role_id"`
DefaultListRoleID int `koanf:"default_list_role_id"`
} `koanf:"oidc"`
Captcha struct {
Altcha struct {
Enabled bool `koanf:"enabled"`
Complexity int `koanf:"complexity"`
} `koanf:"altcha"`
HCaptcha struct {
Enabled bool `koanf:"enabled"`
Key string `koanf:"key"`
Secret string `koanf:"secret"`
} `koanf:"hcaptcha"`
} `koanf:"captcha"`
CorsOrigins []string `koanf:"cors_origins"`
} `koanf:"security"`
Appearance struct {
AdminCSS []byte `koanf:"admin.custom_css"`
AdminJS []byte `koanf:"admin.custom_js"`
PublicCSS []byte `koanf:"public.custom_css"`
PublicJS []byte `koanf:"public.custom_js"`
}
HasLegacyUser bool
AssetVersion string
MediaUpload struct {
Provider string
Extensions []string
}
BounceWebhooksEnabled bool
BounceSESEnabled bool
BounceSendgridEnabled bool
BouncePostmarkEnabled bool
BounceForwardemailEnabled bool
PermissionsRaw json.RawMessage
Permissions map[string]struct{}
}
// initFlags initializes the commandline flags into the Koanf instance.
func initFlags(ko *koanf.Koanf) {
f := flag.NewFlagSet("config", flag.ContinueOnError)
f.Usage = func() {
// Register --help handler.
fmt.Println(f.FlagUsages())
os.Exit(0)
}
// Register the commandline flags.
f.StringSlice("config", []string{"config.toml"},
"path to one or more config files (will be merged in order)")
f.Bool("install", false, "setup database (first time)")
f.Bool("idempotent", false, "make --install run only if the database isn't already setup")
f.Bool("upgrade", false, "upgrade database to the current version")
f.Bool("version", false, "show current version of the build")
f.Bool("new-config", false, "generate sample config file (at path given in --config)")
f.String("static-dir", "", "(optional) path to directory with static files")
f.String("i18n-dir", "", "(optional) path to directory with i18n language files")
f.Bool("yes", false, "assume 'yes' to prompts during --install/upgrade")
f.Bool("passive", false, "run in passive mode where campaigns are not processed")
if err := f.Parse(os.Args[1:]); err != nil {
lo.Fatalf("error loading flags: %v", err)
}
if err := ko.Load(posflag.Provider(f, ".", ko), nil); err != nil {
lo.Fatalf("error loading config: %v", err)
}
}
// initConfigFiles loads the given config files into the koanf instance.
func initConfigFiles(files []string, ko *koanf.Koanf) {
for _, f := range files {
lo.Printf("reading config: %s", f)
if err := ko.Load(file.Provider(f), toml.Parser()); err != nil {
if os.IsNotExist(err) {
lo.Fatal("config file not found. If there isn't one yet, run --new-config to generate one.")
}
lo.Fatalf("error loading config from file: %v.", err)
}
}
}
// initFileSystem initializes the stuffbin FileSystem to provide
// access to bundled static assets to the app.
func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem {
var (
// stuffbin real_path:virtual_alias paths to map local assets on disk
// when there an embedded filestystem is not found.
// These paths are joined with appDir.
appFiles = []string{
"./config.toml.sample:config.toml.sample",
"./queries:queries",
"./schema.sql:schema.sql",
"./permissions.json:permissions.json",
}
frontendFiles = []string{
// Admin frontend's static assets accessible at /admin/* during runtime.
// These paths are sourced from frontendDir.
"./:/admin",
}
staticFiles = []string{
// These paths are joined with staticDir.
"./email-templates:static/email-templates",
"./public:/public",
}
i18nFiles = []string{
// These paths are joined with i18nDir.
"./:/i18n",
}
)
// Get the executable's execPath.
execPath, err := os.Executable()
if err != nil {
lo.Fatalf("error getting executable path: %v", err)
}
// Load embedded files in the executable.
hasEmbed := true
fs, err := stuffbin.UnStuff(execPath)
if err != nil {
hasEmbed = false
// Running in local mode. Load local assets into
// the in-memory stuffbin.FileSystem.
lo.Printf("unable to initialize embedded filesystem (%v). Using local filesystem", err)
fs, err = stuffbin.NewLocalFS("/")
if err != nil {
lo.Fatalf("failed to initialize local file for assets: %v", err)
}
}
// If the embed failed, load app and frontend files from the compile-time paths.
files := []string{}
if !hasEmbed {
files = append(files, joinFSPaths(appDir, appFiles)...)
files = append(files, joinFSPaths(frontendDir, frontendFiles)...)
}
// Irrespective of the embeds, if there are user specified static or i18n paths,
// load files from there and override default files (embedded or picked up from CWD).
if !hasEmbed || i18nDir != "" {
if i18nDir == "" {
// Default dir in cwd.
i18nDir = "i18n"
}
lo.Printf("loading i18n files from: %v", i18nDir)
files = append(files, joinFSPaths(i18nDir, i18nFiles)...)
}
if !hasEmbed || staticDir != "" {
if staticDir == "" {
// Default dir in cwd.
staticDir = "static"
} else {
// There is a custom static directory. Any paths that aren't in it, exclude.
sf := []string{}
for _, def := range staticFiles {
s := strings.Split(def, ":")[0]
if _, err := os.Stat(path.Join(staticDir, s)); err == nil {
sf = append(sf, def)
}
}
staticFiles = sf
}
lo.Printf("loading static files from: %v", staticDir)
files = append(files, joinFSPaths(staticDir, staticFiles)...)
}
// No additional files to load.
if len(files) == 0 {
return fs
}
// Load files from disk and overlay into the FS.
fStatic, err := stuffbin.NewLocalFS("/", files...)
if err != nil {
lo.Fatalf("failed reading static files from disk: '%s': %v", staticDir, err)
}
if err := fs.Merge(fStatic); err != nil {
lo.Fatalf("error merging static files: '%s': %v", staticDir, err)
}
return fs
}
// initDB initializes the main DB connection pool and parse and loads the app's
// SQL queries into a prepared query map.
func initDB() *sqlx.DB {
var c struct {
Host string `koanf:"host"`
Port int `koanf:"port"`
User string `koanf:"user"`
Password string `koanf:"password"`
DBName string `koanf:"database"`
SSLMode string `koanf:"ssl_mode"`
Params string `koanf:"params"`
MaxOpen int `koanf:"max_open"`
MaxIdle int `koanf:"max_idle"`
MaxLifetime time.Duration `koanf:"max_lifetime"`
}
if err := ko.Unmarshal("db", &c); err != nil {
lo.Fatalf("error loading db config: %v", err)
}
lo.Printf("connecting to db: %s:%d/%s", c.Host, c.Port, c.DBName)
db, err := sqlx.Connect("postgres",
fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=%s %s", c.Host, c.Port, c.User, c.Password, c.DBName, c.SSLMode, c.Params))
if err != nil {
lo.Fatalf("error connecting to DB: %v", err)
}
db.SetMaxOpenConns(c.MaxOpen)
db.SetMaxIdleConns(c.MaxIdle)
db.SetConnMaxLifetime(c.MaxLifetime)
return db.Unsafe()
}
func readQueries(dir string, fs stuffbin.FileSystem) goyesql.Queries {
out := goyesql.Queries{}
// Glob all the .sql files in the queries directory.
qPath := path.Join(dir, "/*.sql")
files, err := fs.Glob(qPath)
if err != nil {
lo.Fatalf("error reading *.sql query files from %s: %v", qPath, err)
}
// Read and merge queries from all files into one map.
for _, file := range files {
// Read the SQL file.
b, err := fs.Read(file)
if err != nil {
lo.Fatalf("error reading SQL file %s: %v", file, err)
}
// Parse queries in it into a map.
mp, err := goyesql.ParseBytes(b)
if err != nil {
lo.Fatalf("error parsing SQL queries: %v", err)
}
// Merge into the main query map.
maps.Copy(out, mp)
}
return out
}
// prepareQueries queries prepares a query map and returns a *Queries
func prepareQueries(qMap goyesql.Queries, db *sqlx.DB, ko *koanf.Koanf) *models.Queries {
var (
countQuery = "get-campaign-analytics-counts"
linkSel = "*"
)
if ko.Bool("privacy.individual_tracking") {
countQuery = "get-campaign-analytics-unique-counts"
linkSel = "DISTINCT subscriber_id"
}
// These don't exist in the SQL file but are in the queries struct to be prepared.
qMap["get-campaign-view-counts"] = &goyesql.Query{
Query: fmt.Sprintf(qMap[countQuery].Query, "campaign_views"),
Tags: map[string]string{"name": "get-campaign-view-counts"},
}
qMap["get-campaign-click-counts"] = &goyesql.Query{
Query: fmt.Sprintf(qMap[countQuery].Query, "link_clicks"),
Tags: map[string]string{"name": "get-campaign-click-counts"},
}
qMap["get-campaign-link-counts"].Query = fmt.Sprintf(qMap["get-campaign-link-counts"].Query, linkSel)
// Scan and prepare all queries.
var q models.Queries
if err := goyesqlx.ScanToStruct(&q, qMap, db); err != nil {
lo.Fatalf("error preparing SQL queries: %v", err)
}
return &q
}
// initSettings loads settings from the DB into the given Koanf map.
func initSettings(query string, db *sqlx.DB, ko *koanf.Koanf) {
var s types.JSONText
if err := db.Get(&s, query); err != nil {
msg := err.Error()
if err, ok := err.(*pq.Error); ok {
if err.Detail != "" {
msg = fmt.Sprintf("%s. %s", err, err.Detail)
}
}
lo.Fatalf("error reading settings from DB: %s", msg)
}
// Setting keys are dot separated, eg: app.favicon_url. Unflatten them into
// nested maps {app: {favicon_url}}.
var out map[string]any
if err := json.Unmarshal(s, &out); err != nil {
lo.Fatalf("error unmarshalling settings from DB: %v", err)
}
if err := ko.Load(confmap.Provider(out, "."), nil); err != nil {
lo.Fatalf("error parsing settings from DB: %v", err)
}
}
func initUrlConfig(ko *koanf.Koanf) *UrlConfig {
root := strings.TrimSuffix(ko.String("app.root_url"), "/")
return &UrlConfig{
RootURL: root,
LogoURL: ko.String("app.logo_url"),
FaviconURL: ko.String("app.favicon_url"),
LoginURL: path.Join(uriAdmin, "/login"),
// Static URLS.
// url.com/subscription/{campaign_uuid}/{subscriber_uuid}
UnsubURL: fmt.Sprintf("%s/subscription/%%s/%%s", root),
// url.com/subscription/optin/{subscriber_uuid}
OptinURL: fmt.Sprintf("%s/subscription/optin/%%s?%%s", root),
// url.com/link/{campaign_uuid}/{subscriber_uuid}/{link_uuid}
LinkTrackURL: fmt.Sprintf("%s/link/%%s/%%s/%%s", root),
// url.com/link/{campaign_uuid}/{subscriber_uuid}
MessageURL: fmt.Sprintf("%s/campaign/%%s/%%s", root),
// url.com/archive
ArchiveURL: root + "/archive",
// url.com/campaign/{campaign_uuid}/{subscriber_uuid}/px.png
ViewTrackURL: fmt.Sprintf("%s/campaign/%%s/%%s/px.png", root),
}
}
// initConstConfig initializes the app's global constants from the given koanf instance.
func initConstConfig(ko *koanf.Koanf) *Config {
// Read constants.
var c Config
if err := ko.Unmarshal("app", &c); err != nil {
lo.Fatalf("error loading app config: %v", err)
}
if err := ko.Unmarshal("privacy", &c.Privacy); err != nil {
lo.Fatalf("error loading app.privacy config: %v", err)
}
if err := ko.Unmarshal("security", &c.Security); err != nil {
lo.Fatalf("error loading app.security config: %v", err)
}
if err := ko.UnmarshalWithConf("appearance", &c.Appearance, koanf.UnmarshalConf{FlatPaths: true}); err != nil {
lo.Fatalf("error loading app.appearance config: %v", err)
}
c.Lang = ko.String("app.lang")
c.Privacy.Exportable = koanfmaps.StringSliceToLookupMap(ko.Strings("privacy.exportable"))
c.MediaUpload.Provider = ko.String("upload.provider")
c.MediaUpload.Extensions = ko.Strings("upload.extensions")
c.Privacy.DomainBlocklist = ko.Strings("privacy.domain_blocklist")
c.Privacy.DomainAllowlist = ko.Strings("privacy.domain_allowlist")
c.BounceWebhooksEnabled = ko.Bool("bounce.webhooks_enabled")
c.BounceSESEnabled = ko.Bool("bounce.ses_enabled")
c.BounceSendgridEnabled = ko.Bool("bounce.sendgrid_enabled")
c.BouncePostmarkEnabled = ko.Bool("bounce.postmark.enabled")
c.BounceForwardemailEnabled = ko.Bool("bounce.forwardemail.enabled")
c.HasLegacyUser = ko.Exists("app.admin_username") || ko.Exists("app.admin_password")
b := md5.Sum([]byte(time.Now().String()))
c.AssetVersion = fmt.Sprintf("%x", b)[0:10]
pm, err := fs.Read("/permissions.json")
if err != nil {
lo.Fatalf("error reading permissions file: %v", err)
}
c.PermissionsRaw = pm
// Make a lookup map of permissions.
permGroups := []struct {
Group string `json:"group"`
Permissions []string `json:"permissions"`
}{}
if err := json.Unmarshal(pm, &permGroups); err != nil {
lo.Fatalf("error loading permissions file: %v", err)
}
c.Permissions = map[string]struct{}{}
for _, group := range permGroups {
for _, g := range group.Permissions {
c.Permissions[g] = struct{}{}
}
}
return &c
}
// initI18n initializes a new i18n instance with the selected language map
// loaded from the filesystem. English is a loaded first as the default map
// and then the selected language is loaded on top of it so that if there are
// missing translations in it, the default English translations show up.
func initI18n(lang string, fs stuffbin.FileSystem) *i18n.I18n {
i, ok, err := getI18nLang(lang, fs)
if err != nil {
if ok {
lo.Println(err)
} else {
lo.Fatal(err)
}
}
return i
}
// initCore initializes the CRUD DB core .
func initCore(fnNotify func(sub models.Subscriber, listIDs []int) (int, error), queries *models.Queries, db *sqlx.DB, i *i18n.I18n, ko *koanf.Koanf) *core.Core {
opt := &core.Opt{
Constants: core.Constants{
SendOptinConfirmation: ko.Bool("app.send_optin_confirmation"),
CacheSlowQueries: ko.Bool("app.cache_slow_queries"),
},
Queries: queries,
DB: db,
I18n: i,
Log: lo,
}
// Load bounce config.
if err := ko.Unmarshal("bounce.actions", &opt.Constants.BounceActions); err != nil {
lo.Fatalf("error unmarshalling bounce config: %v", err)
}
// Initialize the CRUD core.
return core.New(opt, &core.Hooks{
SendOptinConfirmation: fnNotify,
})
}
// initCampaignManager initializes the campaign manager.
func initCampaignManager(msgrs []manager.Messenger, q *models.Queries, u *UrlConfig, co *core.Core, md media.Store, i *i18n.I18n, ko *koanf.Koanf) *manager.Manager {
if ko.Bool("passive") {
lo.Println("running in passive mode. won't process campaigns.")
}
mgr := manager.New(manager.Config{
BatchSize: ko.Int("app.batch_size"),
Concurrency: ko.Int("app.concurrency"),
MessageRate: ko.Int("app.message_rate"),
MaxSendErrors: ko.Int("app.max_send_errors"),
FromEmail: ko.String("app.from_email"),
IndividualTracking: ko.Bool("privacy.individual_tracking"),
DisableTracking: ko.Bool("privacy.disable_tracking"),
UnsubURL: u.UnsubURL,
OptinURL: u.OptinURL,
LinkTrackURL: u.LinkTrackURL,
ViewTrackURL: u.ViewTrackURL,
MessageURL: u.MessageURL,
ArchiveURL: u.ArchiveURL,
RootURL: u.RootURL,
UnsubHeader: ko.Bool("privacy.unsubscribe_header"),
SlidingWindow: ko.Bool("app.message_sliding_window"),
SlidingWindowDuration: ko.Duration("app.message_sliding_window_duration"),
SlidingWindowRate: ko.Int("app.message_sliding_window_rate"),
ScanInterval: time.Second * 5,
ScanCampaigns: !ko.Bool("passive"),
}, newManagerStore(q, co, md), i, lo)
// Attach all messengers to the campaign manager.
for _, m := range msgrs {
mgr.AddMessenger(m)
}
return mgr
}
// initTxTemplates initializes and compiles the transactional templates and caches them in-memory.
func initTxTemplates(m *manager.Manager, co *core.Core) {
tpls, err := co.GetTemplates(models.TemplateTypeTx, false)
if err != nil {
lo.Fatalf("error loading transactional templates: %v", err)
}
for _, t := range tpls {
tpl := t
if err := tpl.Compile(m.GenericTemplateFuncs()); err != nil {
lo.Printf("error compiling transactional template %d: %v", tpl.ID, err)
continue
}
m.CacheTpl(tpl.ID, &tpl)
}
}
// initImporter initializes the bulk subscriber importer.
func initImporter(q *models.Queries, db *sqlx.DB, core *core.Core, i *i18n.I18n, ko *koanf.Koanf) *subimporter.Importer {
return subimporter.New(
subimporter.Options{
DomainBlocklist: ko.Strings("privacy.domain_blocklist"),
DomainAllowlist: ko.Strings("privacy.domain_allowlist"),
UpsertStmt: q.UpsertSubscriber.Stmt,
BlocklistStmt: q.UpsertBlocklistSubscriber.Stmt,
UpdateListDateStmt: q.UpdateListsDate.Stmt,
// Hook for triggering admin notifications and refreshing stats materialized
// views after a successful import.
PostCB: func(subject string, data any) error {
// Refresh cached subscriber counts and stats.
core.RefreshMatViews(true)
// Send admin notification.
notifs.NotifySystem(subject, notifs.TplImport, data, nil)
return nil
},
}, db.DB, i)
}
// initSMTPMessenger initializes the combined and individual SMTP messengers.
func initSMTPMessengers() []manager.Messenger {
var (
servers = []email.Server{}
out = []manager.Messenger{}
)
// Load the config for multiple SMTP servers.
for _, item := range ko.Slices("smtp") {
if !item.Bool("enabled") {
continue
}
// Read the SMTP config.
var s email.Server
if err := item.UnmarshalWithConf("", &s, koanf.UnmarshalConf{Tag: "json"}); err != nil {
lo.Fatalf("error reading SMTP config: %v", err)
}
servers = append(servers, s)
lo.Printf("initialized email (SMTP) messenger: %s@%s", item.String("username"), item.String("host"))
// If the server has a name, initialize it as a standalone e-mail messenger
// allowing campaigns to select individual SMTPs. In the UI and config, it'll appear as `email / $name`.
if s.Name != "" {
msgr, err := email.New(s.Name, s)
if err != nil {
lo.Fatalf("error initializing e-mail messenger: %v", err)
}
out = append(out, msgr)
}
}
// Initialize the 'email' messenger with all SMTP servers.
msgr, err := email.New(email.MessengerName, servers...)
if err != nil {
lo.Fatalf("error initializing e-mail messenger: %v", err)
}
// If it's just one server, return the default "email" messenger.
if len(servers) == 1 {
return []manager.Messenger{msgr}
}
// If there are multiple servers, prepend the group "email" to be the first one.
out = append([]manager.Messenger{msgr}, out...)
return out
}
// initPostbackMessengers initializes and returns all the enabled
// HTTP postback messenger backends.
func initPostbackMessengers(ko *koanf.Koanf) []manager.Messenger {
items := ko.Slices("messengers")
if len(items) == 0 {
return nil
}
var out []manager.Messenger
for _, item := range items {
if !item.Bool("enabled") {
continue
}
// Read the Postback server config.
var (
name = item.String("name")
o postback.Options
)
if err := item.UnmarshalWithConf("", &o, koanf.UnmarshalConf{Tag: "json"}); err != nil {
lo.Fatalf("error reading Postback config: %v", err)
}
// Initialize the Messenger.
p, err := postback.New(o)
if err != nil {
lo.Fatalf("error initializing Postback messenger %s: %v", name, err)
}
out = append(out, p)
lo.Printf("loaded Postback messenger: %s", name)
}
return out
}
// initMediaStore initializes Upload manager with a custom backend.
func initMediaStore(ko *koanf.Koanf) media.Store {
switch provider := ko.String("upload.provider"); provider {
case "s3":
var o s3.Opt
ko.Unmarshal("upload.s3", &o)
o.RootURL = ko.String("app.root_url")
up, err := s3.NewS3Store(o)
if err != nil {
lo.Fatalf("error initializing s3 upload provider %s", err)
}
lo.Println("media upload provider: s3")
return up
case "filesystem":
var o filesystem.Opts
ko.Unmarshal("upload.filesystem", &o)
o.RootURL = ko.String("app.root_url")
o.UploadPath = filepath.Clean(o.UploadPath)
o.UploadURI = filepath.Clean(o.UploadURI)
up, err := filesystem.New(o)
if err != nil {
lo.Fatalf("error initializing filesystem upload provider %s", err)
}
lo.Println("media upload provider: filesystem")
return up
default:
lo.Fatalf("unknown provider. select filesystem or s3")
}
return nil
}
// initNotifs initializes the notifier with the system e-mail templates.
func initNotifs(fs stuffbin.FileSystem, i *i18n.I18n, em *email.Emailer, u *UrlConfig, ko *koanf.Koanf) {
tpls, err := stuffbin.ParseTemplatesGlob(initTplFuncs(i, u), fs, "/static/email-templates/*.html")
if err != nil {
lo.Fatalf("error parsing e-mail notif templates: %v", err)
}
// Read the notification templates.
html, err := fs.Read("/static/email-templates/base.html")
if err != nil {
lo.Fatalf("error reading static/email-templates/base.html: %v", err)
}
// Determine whether the notification templates are HTML or plaintext.
// Copy the first few (arbitrary) bytes of the template and check if has the tag.
ln := min(len(html), 256)
h := make([]byte, ln)
copy(h, html[0:ln])
contentType := models.CampaignContentTypeHTML
if !bytes.Contains(bytes.ToLower(h), []byte(" 0 {
c.Start()
}
}
// awaitReload waits for a SIGHUP signal to reload the app. Every setting change on the UI causes a reload.
func awaitReload(sigChan chan os.Signal, closerWait chan bool, closer func()) chan bool {
// The blocking signal handler that main() waits on.
out := make(chan bool)
// Respawn a new process and exit the running one.
respawn := func() {
if err := syscall.Exec(os.Args[0], os.Args, os.Environ()); err != nil {
lo.Fatalf("error spawning process: %v", err)
}
os.Exit(0)
}
// Listen for reload signal.
go func() {
for range sigChan {
lo.Println("reloading on signal ...")
go closer()
select {
case <-closerWait:
// Wait for the closer to finish.
respawn()
case <-time.After(time.Second * 3):
// Or timeout and force close.
respawn()
}
}
}()
return out
}
// initTplFuncs returns a generic template func map with custom template
// functions and sprig template functions.
func initTplFuncs(i *i18n.I18n, u *UrlConfig) template.FuncMap {
funcs := template.FuncMap{
"RootURL": func() string {
return u.RootURL
},
"LogoURL": func() string {
return u.LogoURL
},
"Date": func(layout string) string {
if layout == "" {
layout = time.ANSIC
}
return time.Now().Format(layout)
},
"L": func() *i18n.I18n {
return i
},
"Safe": func(safeHTML string) template.HTML {
return template.HTML(safeHTML)
},
}
// Copy spring functions.
sprigFuncs := sprig.GenericFuncMap()
delete(sprigFuncs, "env")
delete(sprigFuncs, "expandenv")
delete(sprigFuncs, "getHostByName")
maps.Copy(funcs, sprigFuncs)
return funcs
}
// initAuth initializes the auth module with the given DB connection and
func initAuth(co *core.Core, db *sql.DB, ko *koanf.Koanf) (bool, *auth.Auth) {
var oidcCfg auth.OIDCConfig
// If OIDC is enabled, set up the OIDC config.
if ko.Bool("security.oidc.enabled") {
oidcCfg = auth.OIDCConfig{
Enabled: true,
ProviderURL: ko.String("security.oidc.provider_url"),
ClientID: ko.String("security.oidc.client_id"),
ClientSecret: ko.String("security.oidc.client_secret"),
AutoCreateUsers: ko.Bool("security.oidc.auto_create_users"),
DefaultUserRoleID: ko.Int("security.oidc.default_user_role_id"),
DefaultListRoleID: ko.Int("security.oidc.default_list_role_id"),
RedirectURL: fmt.Sprintf("%s/auth/oidc", strings.TrimRight(ko.String("app.root_url"), "/")),
}
}
// Setup the sessio manager callbacks for getting and setting cookies.
cb := &auth.Callbacks{
GetCookie: func(name string, r any) (*http.Cookie, error) {
c := r.(echo.Context)
cookie, err := c.Cookie(name)
return cookie, err
},
SetCookie: func(cookie *http.Cookie, w any) error {
c := w.(echo.Context)
cookie.SameSite = http.SameSiteLaxMode
c.SetCookie(cookie)
return nil
},
GetUser: func(id int) (auth.User, error) {
return co.GetUser(id, "", "")
},
}
// Initiaize the auth module.
a, err := auth.New(auth.Config{OIDC: oidcCfg}, db, cb, lo)
if err != nil {
lo.Fatalf("error initializing auth: %v", err)
}
// Cache all API users in-memory for token auth.
hasUsers, err := cacheUsers(co, a)
if err != nil {
lo.Fatalf("error loading API users to cache: %v", err)
}
// If the legacy username+password is set in the TOML file, use that as an API
// access token in the auth module to preserve backwards compatibility for existing
// API integrations. The presence of these values show a red banner on the admin UI
// prompting the creation of new API credentials and the removal of values from
// the TOML config.
var (
username = ko.String("app.admin_username")
password = ko.String("app.admin_password")
)
if len(username) > 2 && len(password) > 6 {
u := auth.User{
Username: username,
Password: null.String{Valid: true, String: password},
PasswordLogin: true,
HasPassword: true,
Status: auth.UserStatusEnabled,
Type: auth.UserTypeAPI,
}
u.UserRole.ID = auth.SuperAdminRoleID
a.CacheAPIUser(u)
lo.Println(`WARNING: Remove the admin_username and admin_password fields from the TOML configuration file. If you are using APIs, create and use new credentials. Users are now managed via the Admin -> Settings -> Users dashboard.`)
}
return hasUsers, a
}
// joinFSPaths joins the given paths with the root path and returns the full paths.
func joinFSPaths(root string, paths []string) []string {
out := make([]string, 0, len(paths))
for _, p := range paths {
// real_path:stuffbin_alias
f := strings.Split(p, ":")
out = append(out, path.Join(root, f[0])+":"+f[1])
}
return out
}
================================================
FILE: cmd/install.go
================================================
package main
import (
"encoding/json"
"fmt"
"os"
"strings"
"github.com/gofrs/uuid/v5"
"github.com/jmoiron/sqlx"
"github.com/knadh/listmonk/internal/auth"
"github.com/knadh/listmonk/internal/utils"
"github.com/knadh/listmonk/models"
"github.com/knadh/stuffbin"
"github.com/lib/pq"
null "gopkg.in/volatiletech/null.v6"
)
// install runs the first time setup of setting up the database.
func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempotent bool) {
qMap := readQueries(queryFilePath, fs)
fmt.Println("")
if !idempotent {
fmt.Println("** first time installation **")
fmt.Printf("** IMPORTANT: This will wipe existing listmonk tables and types in the DB '%s' **",
ko.String("db.database"))
} else {
fmt.Println("** first time (idempotent) installation **")
}
fmt.Println("")
if prompt {
var ok string
fmt.Print("continue (y/N)? ")
if _, err := fmt.Scanf("%s", &ok); err != nil {
lo.Fatalf("error reading value from terminal: %v", err)
}
if strings.ToLower(ok) != "y" {
fmt.Println("install cancelled.")
return
}
}
// If idempotence is on, check if the DB is already setup.
if idempotent {
if _, err := db.Exec("SELECT count(*) FROM settings"); err != nil {
// If "settings" doesn't exist, assume it's a fresh install.
if pqErr, ok := err.(*pq.Error); ok && pqErr.Code != "42P01" {
lo.Fatalf("error checking existing DB schema: %v", err)
}
} else {
lo.Println("skipping install as database appears to be already setup")
os.Exit(0)
}
}
// Migrate the tables.
if err := installSchema(lastVer, db, fs); err != nil {
lo.Fatalf("error migrating DB schema: %v", err)
}
// Load the queries.
q := prepareQueries(qMap, db, ko)
// Sample list.
defList, optinList := installLists(q)
// Sample subscribers.
installSubs(defList, optinList, q)
// Templates.
campTplID, archiveTplID := installTemplates(q)
// Sample campaign.
installCampaign(campTplID, archiveTplID, q)
// Setup admin user optionally.
var (
user = os.Getenv("LISTMONK_ADMIN_USER")
password = os.Getenv("LISTMONK_ADMIN_PASSWORD")
apiUser = os.Getenv("LISTMONK_ADMIN_API_USER")
hasUser = false
)
// Admin user.
if user != "" && password != "" {
if len(user) < 3 || len(password) < 8 {
lo.Fatal("LISTMONK_ADMIN_USER should be min 3 chars and LISTMONK_ADMIN_PASSWORD should be min 8 chars")
}
lo.Printf("creating superadmin user '%s'", user)
hasUser = true
} else {
lo.Printf("no superadmin user created. Visit webpage to create user.")
}
// API User.
if apiUser != "" {
if !hasUser {
lo.Fatal("LISTMONK_ADMIN_API_USER requires LISTMONK_ADMIN_USER and LISTMONK_ADMIN_PASSWORD to be set")
}
if len(apiUser) < 3 {
lo.Fatal("LISTMONK_ADMIN_API_USER should be min 3 chars")
}
lo.Printf("creating superadmin API user '%s'", apiUser)
}
if hasUser {
installUser(user, password, apiUser, q)
}
lo.Printf("setup complete")
lo.Printf(`run the program and access the dashboard at %s`, ko.MustString("app.address"))
}
// installSchema executes the SQL schema and creates the necessary tables and types.
func installSchema(curVer string, db *sqlx.DB, fs stuffbin.FileSystem) error {
q, err := fs.Read("/schema.sql")
if err != nil {
return err
}
if _, err := db.Exec(string(q)); err != nil {
return err
}
// Insert the current migration version.
return recordMigrationVersion(curVer, db)
}
func installLists(q *models.Queries) (int, int) {
var (
defList int
optinList int
)
if err := q.CreateList.Get(&defList,
uuid.Must(uuid.NewV4()),
"Default list",
models.ListTypePrivate,
models.ListOptinSingle,
models.ListStatusActive,
pq.StringArray{"test"},
"",
); err != nil {
lo.Fatalf("error creating list: %v", err)
}
if err := q.CreateList.Get(&optinList, uuid.Must(uuid.NewV4()),
"Opt-in list",
models.ListTypePublic,
models.ListOptinDouble,
models.ListStatusActive,
pq.StringArray{"test"},
"",
); err != nil {
lo.Fatalf("error creating list: %v", err)
}
return defList, optinList
}
func installSubs(defListID, optinListID int, q *models.Queries) {
// Sample subscriber.
if _, err := q.UpsertSubscriber.Exec(
uuid.Must(uuid.NewV4()),
"john@example.com",
"John Doe",
`{"type": "known", "good": true, "city": "Bengaluru"}`,
pq.Int64Array{int64(defListID)},
models.SubscriptionStatusUnconfirmed,
true, true); err != nil {
lo.Fatalf("Error creating subscriber: %v", err)
}
if _, err := q.UpsertSubscriber.Exec(
uuid.Must(uuid.NewV4()),
"anon@example.com",
"Anon Doe",
`{"type": "unknown", "good": true, "city": "Bengaluru"}`,
pq.Int64Array{int64(optinListID)},
models.SubscriptionStatusUnconfirmed,
true, true); err != nil {
lo.Fatalf("error creating subscriber: %v", err)
}
}
func installTemplates(q *models.Queries) (int, int) {
// Default campaign template.
campTpl, err := fs.Get("/static/email-templates/default.tpl")
if err != nil {
lo.Fatalf("error reading default e-mail template: %v", err)
}
var campTplID int
if err := q.CreateTemplate.Get(&campTplID, "Default campaign template", models.TemplateTypeCampaign, "", campTpl.ReadBytes(), nil); err != nil {
lo.Fatalf("error creating default campaign template: %v", err)
}
if _, err := q.SetDefaultTemplate.Exec(campTplID); err != nil {
lo.Fatalf("error setting default template: %v", err)
}
// Default campaign archive template.
archiveTpl, err := fs.Get("/static/email-templates/default-archive.tpl")
if err != nil {
lo.Fatalf("error reading default archive template: %v", err)
}
var archiveTplID int
if err := q.CreateTemplate.Get(&archiveTplID, "Default archive template", models.TemplateTypeCampaign, "", archiveTpl.ReadBytes(), nil); err != nil {
lo.Fatalf("error creating default campaign template: %v", err)
}
// Sample tx template.
txTpl, err := fs.Get("/static/email-templates/sample-tx.tpl")
if err != nil {
lo.Fatalf("error reading default e-mail template: %v", err)
}
if _, err := q.CreateTemplate.Exec("Sample transactional template", models.TemplateTypeTx, "Welcome {{ .Subscriber.Name }}", txTpl.ReadBytes(), nil); err != nil {
lo.Fatalf("error creating sample transactional template: %v", err)
}
// Sample visual campaign template.
visualTpl, err := fs.Get("/static/email-templates/default-visual.tpl")
if err != nil {
lo.Fatalf("error reading default visual template: %v", err)
}
visualSrc, err := fs.Get("/static/email-templates/default-visual.json")
if err != nil {
lo.Fatalf("error reading default visual template json: %v", err)
}
if _, err := q.CreateTemplate.Exec("Sample visual template", models.TemplateTypeCampaignVisual, "", visualTpl.ReadBytes(), visualSrc.ReadBytes()); err != nil {
lo.Fatalf("error creating default campaign template: %v", err)
}
return campTplID, archiveTplID
}
func installCampaign(campTplID, archiveTplID int, q *models.Queries) {
// Sample campaign.
if _, err := q.CreateCampaign.Exec(uuid.Must(uuid.NewV4()),
models.CampaignTypeRegular,
"Test campaign",
"Welcome to listmonk",
"No Reply Hi {{ .Subscriber.FirstName }}!
This is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.
Here is a tracked link.
Use the link icon in the editor toolbar or when writing raw HTML or Markdown, simply suffix @TrackLink to the end of a URL to turn it into a tracking link. Example:
<a href="https://listmonk.app@TrackLink"></a>
For help, refer to the documentation.
`, nil, "richtext", nil, json.RawMessage("[]"), json.RawMessage("{}"), pq.StringArray{"test-campaign"}, emailMsgr, campTplID, pq.Int64Array{1}, false, "welcome-to-listmonk", archiveTplID, `{"name": "Subscriber"}`, nil, nil, ); err != nil { lo.Fatalf("error creating sample campaign: %v", err) } } // recordMigrationVersion inserts the given version (of DB migration) into the // `migrations` array in the settings table. func recordMigrationVersion(ver string, db *sqlx.DB) error { _, err := db.Exec(fmt.Sprintf(`INSERT INTO settings (key, value) VALUES('migrations', '["%s"]'::JSONB) ON CONFLICT (key) DO UPDATE SET value = settings.value || EXCLUDED.value`, ver)) return err } func newConfigFile(path string) error { if _, err := os.Stat(path); !os.IsNotExist(err) { return fmt.Errorf("error creating %s: %v", path, err) } // Initialize the static file system into which all // required static assets (.sql, .js files etc.) are loaded. fs := initFS(appDir, "", "", "") b, err := fs.Read("config.toml.sample") if err != nil { return fmt.Errorf("error reading sample config (is binary stuffed?): %v", err) } return os.WriteFile(path, b, 0644) } // checkSchema checks if the DB schema is installed. func checkSchema(db *sqlx.DB) (bool, error) { if _, err := db.Exec(`SELECT id FROM templates LIMIT 1`); err != nil { if isTableNotExistErr(err) { return false, nil } return false, err } return true, nil } func installUser(username, password, apiUsername string, q *models.Queries) { consts := initConstConfig(ko) // Super Admin role gets all permissions. perms := []string{} for p := range consts.Permissions { perms = append(perms, p) } // Create the Super Admin role in the DB. var role auth.Role if err := q.CreateRole.Get(&role, "Super Admin", auth.RoleTypeUser, pq.Array(perms)); err != nil { lo.Fatalf("error creating super admin role: %v", err) } // Create the admin user. if _, err := q.CreateUser.Exec(username, true, password, username+"@listmonk", username, auth.RoleTypeUser, role.ID, nil, auth.UserStatusEnabled); err != nil { lo.Fatalf("error creating superadmin user: %v", err) } // Create the admin API user. if apiUsername != "" { // Generate a random API token. tk, err := utils.GenerateRandomString(32) if err != nil { lo.Fatalf("error generating API token: %v", err) } var ( email = null.String{String: apiUsername + "@api", Valid: true} password = null.String{String: tk, Valid: true} ) if _, err := q.CreateUser.Exec(apiUsername, false, password, email, apiUsername, auth.UserTypeAPI, role.ID, nil, auth.UserStatusEnabled); err != nil { lo.Fatalf("error creating superadmin API user: %v", err) } // Print the token to stdout so that it can be grepped out. lo.Println("writing API token LISTMONK_ADMIN_API_TOKEN to stderr") fmt.Fprintf(os.Stderr, "export LISTMONK_ADMIN_API_TOKEN=\"%s\"\n", tk) } } ================================================ FILE: cmd/lists.go ================================================ package main import ( "net/http" "strconv" "strings" "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" ) // GetLists retrieves lists with additional metadata like subscriber counts. func (a *App) GetLists(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Get the list IDs (or blanket permission) the user has access to. hasAllPerm, permittedIDs := user.GetPermittedLists(auth.PermTypeGet) // Minimal query simply returns the list of all lists without JOIN subscriber counts. This is fast. minimal, _ := strconv.ParseBool(c.FormValue("minimal")) if minimal { status := c.FormValue("status") res, err := a.core.GetLists("", status, hasAllPerm, permittedIDs) if err != nil { return err } if len(res) == 0 { return c.JSON(http.StatusOK, okResp{[]struct{}{}}) } // Meta. total := len(res) out := models.PageResults{ Results: res, Total: total, Page: 1, PerPage: total, } return c.JSON(http.StatusOK, okResp{out}) } // Full list query. var ( query = strings.TrimSpace(c.FormValue("query")) tags = c.QueryParams()["tag"] orderBy = c.FormValue("order_by") typ = c.FormValue("type") optin = c.FormValue("optin") status = c.FormValue("status") order = c.FormValue("order") pg = a.pg.NewFromURL(c.Request().URL.Query()) ) res, total, err := a.core.QueryLists(query, typ, optin, status, tags, orderBy, order, hasAllPerm, permittedIDs, pg.Offset, pg.Limit) if err != nil { return err } out := models.PageResults{ Query: query, Results: res, Total: total, Page: pg.Page, PerPage: pg.PerPage, } return c.JSON(http.StatusOK, okResp{out}) } // GetList retrieves a single list by id. // It's permission checked by the listPerm middleware. func (a *App) GetList(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Check if the user has access to the list. id := getID(c) if err := user.HasListPerm(auth.PermTypeGet, id); err != nil { return err } // Get the list from the DB. out, err := a.core.GetList(id, "") if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // CreateList handles list creation. func (a *App) CreateList(c echo.Context) error { l := models.List{} if err := c.Bind(&l); err != nil { return err } // Validate. if !strHasLen(l.Name, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("lists.invalidName")) } out, err := a.core.CreateList(l) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // UpdateList handles list modification. // It's permission checked by the listPerm middleware. func (a *App) UpdateList(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Check if the user has access to the list. id := getID(c) if err := user.HasListPerm(auth.PermTypeManage, id); err != nil { return err } // Incoming params. var l models.List if err := c.Bind(&l); err != nil { return err } // Validate. if !strHasLen(l.Name, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("lists.invalidName")) } // Update the list in the DB. out, err := a.core.UpdateList(id, l) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // DeleteList deletes a single list by ID. func (a *App) DeleteList(c echo.Context) error { id := getID(c) // Check if the user has manage permission for the list. user := auth.GetUser(c) if err := user.HasListPerm(auth.PermTypeManage, id); err != nil { return err } // Delete the list from the DB. // Pass getAll=true since we've already verified permissions above. if err := a.core.DeleteLists([]int{id}, "", true, nil); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteLists deletes multiple lists by IDs or by query. func (a *App) DeleteLists(c echo.Context) error { user := auth.GetUser(c) var ( ids []int query string all bool ) // Check for IDs in query params. if len(c.Request().URL.Query()["id"]) > 0 { var err error ids, err = parseStringIDs(c.Request().URL.Query()["id"]) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error())) } } else { // Check for query param. query = strings.TrimSpace(c.FormValue("query")) all = c.FormValue("all") == "true" } // Validate that either IDs or query is provided. if len(ids) == 0 && (query == "" && !all) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", "id or query required")) } // For ID deletion, check if the user has manage permission for the specific lists. if len(ids) > 0 { if err := user.HasListPerm(auth.PermTypeManage, ids...); err != nil { return err } // Delete the lists from the DB. // Pass getAll=true since we've already verified permissions above. if err := a.core.DeleteLists(ids, "", true, nil); err != nil { return err } } else { // For query deletion, get the list IDs the user has manage permission for. hasAllPerm, permittedIDs := user.GetPermittedLists(auth.PermTypeManage) // Delete the lists from the DB with permission filtering. if err := a.core.DeleteLists(nil, query, hasAllPerm, permittedIDs); err != nil { return err } } return c.JSON(http.StatusOK, okResp{true}) } ================================================ FILE: cmd/main.go ================================================ package main import ( "context" "fmt" "io" "log" "os" "os/signal" "strings" "sync" "syscall" "time" "github.com/jmoiron/sqlx" "github.com/knadh/koanf/providers/env" "github.com/knadh/koanf/v2" "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/internal/bounce" "github.com/knadh/listmonk/internal/buflog" "github.com/knadh/listmonk/internal/captcha" "github.com/knadh/listmonk/internal/core" "github.com/knadh/listmonk/internal/events" "github.com/knadh/listmonk/internal/i18n" "github.com/knadh/listmonk/internal/manager" "github.com/knadh/listmonk/internal/media" "github.com/knadh/listmonk/internal/messenger/email" "github.com/knadh/listmonk/internal/subimporter" "github.com/knadh/listmonk/models" "github.com/knadh/paginator" "github.com/knadh/stuffbin" ) // App contains the "global" shared components, controllers and fields. type App struct { cfg *Config urlCfg *UrlConfig fs stuffbin.FileSystem db *sqlx.DB queries *models.Queries core *core.Core manager *manager.Manager messengers []manager.Messenger emailMsgr manager.Messenger importer *subimporter.Importer auth *auth.Auth media media.Store bounce *bounce.Manager captcha *captcha.Captcha i18n *i18n.I18n pg *paginator.Paginator events *events.Events log *log.Logger bufLog *buflog.BufLog about about fnOptinNotify func(models.Subscriber, []int) (int, error) // Channel for passing reload signals. chReload chan os.Signal // Global variable that stores the state indicating that a restart is required // after a settings update. needsRestart bool // First time installation with no user records in the DB. Needs user setup. needsUserSetup bool // Global state that stores data on an available remote update. update *AppUpdate sync.Mutex } var ( // Buffered log writer for storing N lines of log entries for the UI. evStream = events.New() bufLog = buflog.New(5000) lo = log.New(io.MultiWriter(os.Stdout, bufLog, evStream.ErrWriter()), "", log.Ldate|log.Ltime|log.Lmicroseconds|log.Lshortfile) ko = koanf.New(".") fs stuffbin.FileSystem db *sqlx.DB queries *models.Queries // Compile-time variables. buildString string versionString string // If these are set in build ldflags and static assets (*.sql, config.toml.sample. ./frontend) // are not embedded (in make dist), these paths are looked up. The default values before, when not // overridden by build flags, are relative to the CWD at runtime. appDir string = "." frontendDir string = "frontend/dist" ) func init() { // Initialize commandline flags. initFlags(ko) // Display version. if ko.Bool("version") { fmt.Println(buildString) os.Exit(0) } lo.Println(buildString) // Generate new config. if ko.Bool("new-config") { path := ko.Strings("config")[0] if err := newConfigFile(path); err != nil { lo.Println(err) os.Exit(1) } lo.Printf("generated %s. Edit and run --install", path) os.Exit(0) } // Load config files to pick up the database settings first. initConfigFiles(ko.Strings("config"), ko) // Load environment variables and merge into the loaded config. // LISTMONK_foo__bar -> foo.bar (double underscore becomes dot for nested config) // LISTMONK_static_dir -> static-dir (top-level keys with underscore become hyphen for CLI flags) if err := ko.Load(env.Provider("LISTMONK_", ".", func(s string) string { key := strings.ToLower(strings.TrimPrefix(s, "LISTMONK_")) key = strings.Replace(key, "__", ".", -1) // Only convert underscore to hyphen for top-level keys (CLI flags like static-dir, i18n-dir) // Nested config keys (containing dots) keep underscores (e.g., db.ssl_mode) if !strings.Contains(key, ".") { key = strings.Replace(key, "_", "-", -1) } return key }), nil); err != nil { lo.Fatalf("error loading config from env: %v", err) } // Connect to the database. db = initDB() // Initialize the embedded filesystem with static assets. fs = initFS(appDir, frontendDir, ko.String("static-dir"), ko.String("i18n-dir")) // Installer mode? This runs before the SQL queries are loaded and prepared // as the installer needs to work on an empty DB. if ko.Bool("install") { // Save the version of the last listed migration. install(migList[len(migList)-1].version, db, fs, !ko.Bool("yes"), ko.Bool("idempotent")) os.Exit(0) } // Is this a nightly build? isNightly := strings.Contains(versionString, "nightly") // Check if the DB schema is installed. if ok, err := checkSchema(db); err != nil { log.Fatalf("error checking schema in DB: %v", err) } else if !ok { lo.Fatal("the database does not appear to be setup. Run --install.") } if ko.Bool("upgrade") { // Even on explicit upgrade runs, for nightly builds, do not record the last // migration version in the DB. lo.Printf("running upgrade...") upgrade(db, fs, !ko.Bool("yes"), !isNightly) os.Exit(0) } // For nightly builds, always auto-run pending migrations without // recording the last version in the DB. Migrations are idempotent, and between // nightly releases, they may change multiple times. if isNightly { lo.Printf("auto-running all migrations for nightly %s since last major version", versionString) upgrade(db, fs, false, false) } else { // Before the queries are prepared, see if there are pending upgrades. checkUpgrade(db) } // Read the SQL queries from the queries file. qMap := readQueries(queryFilePath, fs) // Load settings from DB. if q, ok := qMap["get-settings"]; ok { initSettings(q.Query, db, ko) } // Prepare queries. queries = prepareQueries(qMap, db, ko) } func main() { var ( // Initialize static global config. cfg = initConstConfig(ko) // Initialize static URL config. urlCfg = initUrlConfig(ko) // Initialize i18n language map. i18n = initI18n(ko.MustString("app.lang"), fs) // Initialize the media store. media = initMediaStore(ko) fbOptinNotify = makeOptinNotifyHook(ko.Bool("privacy.unsubscribe_header"), urlCfg, queries, i18n) // Crud core. core = initCore(fbOptinNotify, queries, db, i18n, ko) // Initialize all messengers, SMTP and postback. msgrs = append(initSMTPMessengers(), initPostbackMessengers(ko)...) // Campaign manager. mgr = initCampaignManager(msgrs, queries, urlCfg, core, media, i18n, ko) // Bulk importer. importer = initImporter(queries, db, core, i18n, ko) // Initialize the auth manager. hasUsers, auth = initAuth(core, db.DB, ko) // Initialize the webhook/POP3 bounce processor. bounce *bounce.Manager emailMsgr *email.Emailer chReload = make(chan os.Signal, 1) ) // Initialize the bounce manager that processes bounces from webhooks and // POP3 mailbox scanning. if ko.Bool("bounce.enabled") { bounce = initBounceManager(core.RecordBounce, queries.RecordBounce, lo, ko) } // Assign the default `email` messenger to the app. for _, m := range msgrs { if m.Name() == "email" { emailMsgr = m.(*email.Emailer) } } // Initialize the global admin/sub e-mail notifier. initNotifs(fs, i18n, emailMsgr, urlCfg, ko) // Initialize and cache tx templates in memory. initTxTemplates(mgr, core) // Initialize the bounce manager that processes bounces from webhooks and // POP3 mailbox scanning. if ko.Bool("bounce.enabled") { go bounce.Run() } // Start cronjobs. initCron(core, db) // Start the campaign manager workers. The campaign batches (fetch from DB, push out // messages) get processed at the specified interval. go mgr.Run() // ========================================================================= // Initialize the App{} with all the global shared components, controllers and fields. app := &App{ cfg: cfg, urlCfg: urlCfg, fs: fs, db: db, queries: queries, core: core, manager: mgr, messengers: msgrs, emailMsgr: emailMsgr, importer: importer, auth: auth, media: media, bounce: bounce, captcha: initCaptcha(), i18n: i18n, log: lo, events: evStream, bufLog: bufLog, pg: paginator.New(paginator.Opt{ DefaultPerPage: 20, MaxPerPage: 50, NumPageNums: 10, PageParam: "page", PerPageParam: "per_page", AllowAll: true, }), fnOptinNotify: fbOptinNotify, about: initAbout(queries, db), chReload: chReload, // If there are no users, then the app needs to prompt for new user setup. needsUserSetup: !hasUsers, } // Star the update checker. if ko.Bool("app.check_updates") { go app.checkUpdates(versionString, time.Hour*24) } // Start the app server. srv := initHTTPServer(cfg, urlCfg, i18n, fs, app) // ========================================================================= // Wait for the reload signal with a callback to gracefully shut down resources. // The `wait` channel is passed to awaitReload to wait for the callback to finish // within N seconds, or do a force reload. signal.Notify(chReload, syscall.SIGHUP) closerWait := make(chan bool) <-awaitReload(chReload, closerWait, func() { // Stop the HTTP server. ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second) defer cancel() srv.Shutdown(ctx) // Close the campaign manager. mgr.Close() // Close the DB pool. db.Close() // Close the messenger pool. for _, m := range app.messengers { m.Close() } // Signal the close. closerWait <- true }) } ================================================ FILE: cmd/maintenance.go ================================================ package main import ( "log" "net/http" "time" "github.com/jmoiron/sqlx" "github.com/labstack/echo/v4" ) // GCSubscribers garbage collects (deletes) orphaned or blocklisted subscribers. func (a *App) GCSubscribers(c echo.Context) error { var ( typ = c.Param("type") n int err error ) switch typ { case "blocklisted": n, err = a.core.DeleteBlocklistedSubscribers() case "orphan": n, err = a.core.DeleteOrphanSubscribers() default: err = echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData")) } if err != nil { return err } return c.JSON(http.StatusOK, okResp{struct { Count int `json:"count"` }{n}}) } // GCSubscriptions garbage collects (deletes) orphaned or blocklisted subscribers. func (a *App) GCSubscriptions(c echo.Context) error { // Validate the date. t, err := time.Parse(time.RFC3339, c.FormValue("before_date")) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData")) } // Delete unconfirmed subscriptions from the DB in bulk. n, err := a.core.DeleteUnconfirmedSubscriptions(t) if err != nil { return err } return c.JSON(http.StatusOK, okResp{struct { Count int `json:"count"` }{n}}) } // GCCampaignAnalytics garbage collects (deletes) campaign analytics. func (a *App) GCCampaignAnalytics(c echo.Context) error { t, err := time.Parse(time.RFC3339, c.FormValue("before_date")) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData")) } switch c.Param("type") { case "all": if err := a.core.DeleteCampaignViews(t); err != nil { return err } err = a.core.DeleteCampaignLinkClicks(t) case "views": err = a.core.DeleteCampaignViews(t) case "clicks": err = a.core.DeleteCampaignLinkClicks(t) default: err = echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData")) } if err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // RunDBVacuum runs a full VACUUM on the PostgreSQL database. // VACUUM reclaims storage occupied by dead tuples and updates planner statistics. func RunDBVacuum(db *sqlx.DB, lo *log.Logger) { lo.Println("running database VACUUM ANALYZE") if _, err := db.Exec("VACUUM ANALYZE"); err != nil { lo.Printf("error running VACUUM ANALYZE: %v", err) return } lo.Println("finished database VACUUM ANALYZE") } ================================================ FILE: cmd/manager_store.go ================================================ package main import ( "github.com/gofrs/uuid/v5" "github.com/knadh/listmonk/internal/core" "github.com/knadh/listmonk/internal/manager" "github.com/knadh/listmonk/internal/media" "github.com/knadh/listmonk/models" "github.com/lib/pq" ) // store implements DataSource over the primary // database. type store struct { queries *models.Queries core *core.Core media media.Store } type runningCamp struct { CampaignID int `db:"campaign_id"` CampaignType string `db:"campaign_type"` LastSubscriberID int `db:"last_subscriber_id"` MaxSubscriberID int `db:"max_subscriber_id"` ListID int `db:"list_id"` } func newManagerStore(q *models.Queries, c *core.Core, m media.Store) *store { return &store{ queries: q, core: c, media: m, } } // NextCampaigns retrieves active campaigns ready to be processed excluding // campaigns that are also being processed. Additionally, it takes a map of campaignID:sentCount // of campaigns that are being processed and updates them in the DB. func (s *store) NextCampaigns(currentIDs []int64, sentCounts []int64) ([]*models.Campaign, error) { var out []*models.Campaign err := s.queries.NextCampaigns.Select(&out, pq.Int64Array(currentIDs), pq.Int64Array(sentCounts)) return out, err } // NextSubscribers retrieves a subset of subscribers of a given campaign. // Since batches are processed sequentially, the retrieval is ordered by ID, // and every batch takes the last ID of the last batch and fetches the next // batch above that. func (s *store) NextSubscribers(campID, limit int) ([]models.Subscriber, error) { var camps []runningCamp if err := s.queries.GetRunningCampaign.Select(&camps, campID); err != nil { return nil, err } var listIDs []int for _, c := range camps { listIDs = append(listIDs, c.ListID) } if len(listIDs) == 0 { return nil, nil } var out []models.Subscriber err := s.queries.NextCampaignSubscribers.Select(&out, camps[0].CampaignID, camps[0].CampaignType, camps[0].LastSubscriberID, camps[0].MaxSubscriberID, pq.Array(listIDs), limit) return out, err } // GetCampaign fetches a campaign from the database. func (s *store) GetCampaign(campID int) (*models.Campaign, error) { var out = &models.Campaign{} err := s.queries.GetCampaign.Get(out, campID, nil, nil, "default") return out, err } // UpdateCampaignStatus updates a campaign's status. func (s *store) UpdateCampaignStatus(campID int, status string) error { _, err := s.queries.UpdateCampaignStatus.Exec(campID, status) return err } // UpdateCampaignCounts updates a campaign's status. func (s *store) UpdateCampaignCounts(campID int, toSend int, sent int, lastSubID int) error { _, err := s.queries.UpdateCampaignCounts.Exec(campID, toSend, sent, lastSubID) return err } // GetAttachment fetches a media attachment blob. func (s *store) GetAttachment(mediaID int) (models.Attachment, error) { m, err := s.core.GetMedia(mediaID, "", "", s.media) if err != nil { return models.Attachment{}, err } b, err := s.media.GetBlob(m.URL) if err != nil { return models.Attachment{}, err } return models.Attachment{ Name: m.Filename, Content: b, Header: manager.MakeAttachmentHeader(m.Filename, "base64", m.ContentType), }, nil } // CreateLink registers a URL with a UUID for tracking clicks and returns the UUID. func (s *store) CreateLink(url string) (string, error) { // Create a new UUID for the URL. If the URL already exists in the DB // the UUID in the database is returned. uu, err := uuid.NewV4() if err != nil { return "", err } var out string if err := s.queries.CreateLink.Get(&out, uu, url); err != nil { return "", err } return out, nil } // RecordBounce records a bounce event and returns the bounce count. func (s *store) RecordBounce(b models.Bounce) (int64, int, error) { var res = struct { SubscriberID int64 `db:"subscriber_id"` Num int `db:"num"` }{} err := s.queries.UpdateCampaignStatus.Select(&res, b.SubscriberUUID, b.Email, b.CampaignUUID, b.Type, b.Source, b.Meta) return res.SubscriberID, res.Num, err } // BlocklistSubscriber blocklists a subscriber permanently. func (s *store) BlocklistSubscriber(id int64) error { _, err := s.queries.BlocklistSubscribers.Exec(pq.Int64Array{id}) return err } // DeleteSubscriber deletes a subscriber from the DB. func (s *store) DeleteSubscriber(id int64) error { _, err := s.queries.DeleteSubscribers.Exec(pq.Int64Array{id}) return err } ================================================ FILE: cmd/media.go ================================================ package main import ( "bytes" "mime/multipart" "net/http" "path/filepath" "strings" "github.com/disintegration/imaging" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" ) const ( thumbPrefix = "thumb_" thumbnailSize = 250 ) var ( vectorExts = []string{"svg"} imageExts = []string{"gif", "png", "jpg", "jpeg"} ) // UploadMedia handles media file uploads. func (a *App) UploadMedia(c echo.Context) error { file, err := c.FormFile("file") if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("media.invalidFile", "error", err.Error())) } // Read the file from the HTTP form. src, err := file.Open() if err != nil { return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("media.errorReadingFile", "error", err.Error())) } defer src.Close() var ( // Naive check for content type and extension. ext = strings.TrimPrefix(strings.ToLower(filepath.Ext(file.Filename)), ".") contentType = file.Header.Get("Content-Type") ) // Validate file extension. if !inArray("*", a.cfg.MediaUpload.Extensions) { if ok := inArray(ext, a.cfg.MediaUpload.Extensions); !ok { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("media.unsupportedFileType", "type", ext)) } } // Sanitize the filename. fName := makeFilename(file.Filename) // If the filename already exists in the DB, make it unique by adding a random suffix. if _, err := a.core.GetMedia(0, "", fName, a.media); err == nil { suffix, err := generateRandomString(6) if err != nil { a.log.Printf("error generating random string: %v", err) return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("globals.messages.internalError")) } fName = appendSuffixToFilename(fName, suffix) } // Upload the file to the media store. fName, err = a.media.Put(fName, contentType, src) if err != nil { a.log.Printf("error uploading file: %v", err) return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("media.errorUploading", "error", err.Error())) } // This keeps track of whether the file has to be deleted from the DB and the store // if any of the subsequent steps fail. var ( cleanUp = false thumbfName = "" ) defer func() { if cleanUp { a.media.Delete(fName) if thumbfName != "" { a.media.Delete(thumbfName) } } }() // Thumbnail width and height. var width, height int // Create thumbnail from file for non-vector formats. isImage := inArray(ext, imageExts) if isImage { thumbFile, wi, he, err := processImage(file) if err != nil { cleanUp = true a.log.Printf("error resizing image: %v", err) return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("media.errorResizing", "error", err.Error())) } width = wi height = he // Upload thumbnail. tf, err := a.media.Put(thumbPrefix+fName, contentType, thumbFile) if err != nil { cleanUp = true a.log.Printf("error saving thumbnail: %v", err) return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("media.errorSavingThumbnail", "error", err.Error())) } thumbfName = tf } if inArray(ext, vectorExts) { thumbfName = fName } // Images have metadata. meta := models.JSON{} if isImage { meta = models.JSON{ "width": width, "height": height, } } // Insert the media into the DB. m, err := a.core.InsertMedia(fName, thumbfName, contentType, meta, a.cfg.MediaUpload.Provider, a.media) if err != nil { cleanUp = true return err } return c.JSON(http.StatusOK, okResp{m}) } // GetAllMedia handles retrieval of uploaded media. func (a *App) GetAllMedia(c echo.Context) error { var ( query = c.FormValue("query") pg = a.pg.NewFromURL(c.Request().URL.Query()) ) // Fetch the media items from the DB. res, total, err := a.core.QueryMedia(a.cfg.MediaUpload.Provider, a.media, query, pg.Offset, pg.Limit) if err != nil { return err } out := models.PageResults{ Results: res, Total: total, Page: pg.Page, PerPage: pg.PerPage, } return c.JSON(http.StatusOK, okResp{out}) } // GetMedia handles retrieval of a media item by ID. func (a *App) GetMedia(c echo.Context) error { // Fetch the media item from the DB. id := getID(c) out, err := a.core.GetMedia(id, "", "", a.media) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // DeleteMedia handles deletion of uploaded media. func (a *App) DeleteMedia(c echo.Context) error { // Delete the media from the DB. The query returns the filename. id := getID(c) fname, err := a.core.DeleteMedia(id) if err != nil { return err } // Delete the files from the media store. a.media.Delete(fname) a.media.Delete(thumbPrefix + fname) return c.JSON(http.StatusOK, okResp{true}) } // ServeS3Media serves media files stored in S3 when the public URL is a relative path. func (a *App) ServeS3Media(c echo.Context) error { key := c.Param("filepath") if key == "" { return echo.NewHTTPError(http.StatusBadRequest, "missing media file path") } b, err := a.media.GetBlob(key) if err != nil { a.log.Printf("error fetching media from s3 %s: %v", key, err) return echo.NewHTTPError(http.StatusInternalServerError, "error fetching media") } return c.Stream(http.StatusOK, http.DetectContentType(b), bytes.NewReader(b)) } // processImage reads the image file and returns thumbnail bytes and // the original image's width, and height. func processImage(file *multipart.FileHeader) (*bytes.Reader, int, int, error) { src, err := file.Open() if err != nil { return nil, 0, 0, err } defer src.Close() img, err := imaging.Decode(src) if err != nil { return nil, 0, 0, err } // Encode the image into a byte slice as PNG. var ( thumb = imaging.Resize(img, thumbnailSize, 0, imaging.Lanczos) out bytes.Buffer ) if err := imaging.Encode(&out, thumb, imaging.PNG); err != nil { return nil, 0, 0, err } b := img.Bounds().Max return bytes.NewReader(out.Bytes()), b.X, b.Y, nil } ================================================ FILE: cmd/public.go ================================================ package main import ( "bytes" "database/sql" "fmt" "html/template" "image" "image/png" "io" "net/http" "strconv" "strings" "github.com/knadh/listmonk/internal/captcha" "github.com/knadh/listmonk/internal/i18n" "github.com/knadh/listmonk/internal/manager" "github.com/knadh/listmonk/internal/notifs" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" "github.com/lib/pq" ) const ( tplMessage = "message" ) // tplRenderer wraps a template.tplRenderer for echo. type tplRenderer struct { templates *template.Template SiteName string RootURL string LogoURL string FaviconURL string AssetVersion string EnablePublicSubPage bool EnablePublicArchive bool IndividualTracking bool } // tplData is the data container that is injected // into public templates for accessing data. type tplData struct { SiteName string RootURL string LogoURL string FaviconURL string AssetVersion string EnablePublicSubPage bool EnablePublicArchive bool IndividualTracking bool Data any L *i18n.I18n } type publicTpl struct { Title string Description string } type unsubTpl struct { publicTpl Subscriber models.Subscriber Subscriptions []models.Subscription SubUUID string AllowBlocklist bool AllowExport bool AllowWipe bool AllowPreferences bool ShowManage bool } type optinReq struct { SubUUID string ListUUIDs []string `query:"l" form:"l"` Lists []models.List `query:"-" form:"-"` } type optinTpl struct { publicTpl optinReq } type msgTpl struct { publicTpl MessageTitle string Message string } type subFormTpl struct { publicTpl Lists []models.List Captcha struct { Enabled bool Provider string Key string Complexity int } } var ( pixelPNG = drawTransparentImage(3, 14) ) // Render executes and renders a template for echo. func (t *tplRenderer) Render(w io.Writer, name string, data any, c echo.Context) error { return t.templates.ExecuteTemplate(w, name, tplData{ SiteName: t.SiteName, RootURL: t.RootURL, LogoURL: t.LogoURL, FaviconURL: t.FaviconURL, AssetVersion: t.AssetVersion, EnablePublicSubPage: t.EnablePublicSubPage, EnablePublicArchive: t.EnablePublicArchive, IndividualTracking: t.IndividualTracking, Data: data, L: c.Get("app").(*App).i18n, }) } // GetPublicLists returns the list of public lists with minimal fields // required to submit a subscription. func (a *App) GetPublicLists(c echo.Context) error { // Get all public lists. lists, err := a.core.GetLists(models.ListTypePublic, models.ListStatusActive, true, nil) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.errorFetchingLists")) } type list struct { UUID string `json:"uuid"` Name string `json:"name"` } out := make([]list, 0, len(lists)) for _, l := range lists { out = append(out, list{ UUID: l.UUID, Name: l.Name, }) } return c.JSON(http.StatusOK, out) } // ViewCampaignMessage renders the HTML view of a campaign message. // This is the view the {{ MessageURL }} template tag links to in e-mail campaigns. func (a *App) ViewCampaignMessage(c echo.Context) error { // Get the campaign. campUUID := c.Param("campUUID") camp, err := a.core.GetCampaign(0, campUUID, "") if err != nil { if er, ok := err.(*echo.HTTPError); ok { if er.Code == http.StatusBadRequest { return c.Render(http.StatusNotFound, tplMessage, makeMsgTpl(a.i18n.T("public.notFoundTitle"), "", a.i18n.T("public.campaignNotFound"))) } } return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign"))) } // Get the subscriber. subUUID := c.Param("subUUID") sub, err := a.core.GetSubscriber(0, subUUID, "") if err != nil { if err == sql.ErrNoRows { return c.Render(http.StatusNotFound, tplMessage, makeMsgTpl(a.i18n.T("public.notFoundTitle"), "", a.i18n.T("public.errorFetchingEmail"))) } return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign"))) } // Compile the template. if err := camp.CompileTemplate(a.manager.TemplateFuncs(&camp)); err != nil { a.log.Printf("error compiling template: %v", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign"))) } // Render the message body. msg, err := a.manager.NewCampaignMessage(&camp, sub) if err != nil { a.log.Printf("error rendering message: %v", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingCampaign"))) } return c.HTML(http.StatusOK, string(msg.Body())) } // SubscriptionPage renders the subscription management page and handles unsubscriptions. // This is the view that {{ UnsubscribeURL }} in campaigns link to. func (a *App) SubscriptionPage(c echo.Context) error { var ( subUUID = c.Param("subUUID") showManage, _ = strconv.ParseBool(c.FormValue("manage")) ) // Get the subscriber from the DB. s, err := a.core.GetSubscriber(0, subUUID, "") if err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } // Prepare the public template. out := unsubTpl{ Subscriber: s, SubUUID: subUUID, publicTpl: publicTpl{Title: a.i18n.T("public.unsubscribeTitle")}, AllowBlocklist: a.cfg.Privacy.AllowBlocklist, AllowExport: a.cfg.Privacy.AllowExport, AllowWipe: a.cfg.Privacy.AllowWipe, AllowPreferences: a.cfg.Privacy.AllowPreferences, } // If the subscriber is blocklisted, throw an error. if s.Status == models.SubscriberStatusBlockListed { return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.noSubTitle"), "", a.i18n.Ts("public.blocklisted"))) } // Only show preference management if it's enabled in settings. if a.cfg.Privacy.AllowPreferences { out.ShowManage = showManage // Get the subscriber's lists from the DB to render in the template. subs, err := a.core.GetSubscriptions(0, subUUID, false) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.errorFetchingLists")) } out.Subscriptions = make([]models.Subscription, 0, len(subs)) for _, s := range subs { // Private lists shouldn't be rendered in the template. if s.Type == models.ListTypePrivate { continue } out.Subscriptions = append(out.Subscriptions, s) } } return c.Render(http.StatusOK, "subscription", out) } // SubscriptionPrefs renders the subscription management page and // s unsubscriptions. This is the view that {{ UnsubscribeURL }} in // campaigns link to. func (a *App) SubscriptionPrefs(c echo.Context) error { // Read the form. var req struct { Name string `form:"name" json:"name"` ListUUIDs []string `form:"l" json:"list_uuids"` Blocklist bool `form:"blocklist" json:"blocklist"` Manage bool `form:"manage" json:"manage"` } if err := c.Bind(&req); err != nil { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("globals.messages.invalidData"))) } // Simple unsubscribe. var ( campUUID = c.Param("campUUID") subUUID = c.Param("subUUID") blocklist = a.cfg.Privacy.AllowBlocklist && req.Blocklist ) if !req.Manage || blocklist { if err := a.core.UnsubscribeByCampaign(subUUID, campUUID, blocklist); err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.errorProcessingRequest"))) } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.unsubbedTitle"), "", a.i18n.T("public.unsubbedInfo"))) } // Is preference management enabled? if !a.cfg.Privacy.AllowPreferences { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.invalidFeature"))) } // Manage preferences. req.Name = strings.TrimSpace(req.Name) if req.Name == "" || len(req.Name) > 256 { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("subscribers.invalidName"))) } // Get the subscriber from the DB. sub, err := a.core.GetSubscriber(0, subUUID, "") if err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("globals.messages.pFound", "name", a.i18n.T("globals.terms.subscriber")))) } sub.Name = req.Name // Update the subscriber properties in the DB. if _, err := a.core.UpdateSubscriber(sub.ID, sub); err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.errorProcessingRequest"))) } // Get the subscriber's lists and whatever is not sent in the request (unchecked), // unsubscribe them. reqUUIDs := make(map[string]struct{}) for _, u := range req.ListUUIDs { reqUUIDs[u] = struct{}{} } // Get subscription from teh DB. subs, err := a.core.GetSubscriptions(0, subUUID, false) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.errorFetchingLists")) } // Filter the lists in the request against the subscriptions in the DB. unsubUUIDs := make([]string, 0, len(req.ListUUIDs)) for _, s := range subs { if s.Type == models.ListTypePrivate { continue } if _, ok := reqUUIDs[s.UUID]; !ok { unsubUUIDs = append(unsubUUIDs, s.UUID) } } // Unsubscribe from lists. if err := a.core.UnsubscribeLists([]int{sub.ID}, nil, unsubUUIDs); err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.errorProcessingRequest"))) } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("globals.messages.done"), "", a.i18n.T("public.prefsSaved"))) } // OptinPage renders the double opt-in confirmation page that subscribers // see when they click on the "Confirm subscription" button in double-optin // notifications. func (a *App) OptinPage(c echo.Context) error { var ( subUUID = c.Param("subUUID") confirm, _ = strconv.ParseBool(c.FormValue("confirm")) req optinReq ) if err := c.Bind(&req); err != nil { return err } // Validate list UUIDs if there are incoming UUIDs in the request. if len(req.ListUUIDs) > 0 { for _, l := range req.ListUUIDs { if !reUUID.MatchString(l) { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("globals.messages.invalidUUID"))) } } } // Get the list of subscription lists where the subscriber hasn't confirmed. lists, err := a.core.GetSubscriberLists(0, subUUID, nil, req.ListUUIDs, models.SubscriptionStatusUnconfirmed, "") if err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingLists"))) } // There are no lists to confirm. if len(lists) == 0 { return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.noSubTitle"), "", a.i18n.Ts("public.noSubInfo"))) } // Confirm. if confirm { meta := models.JSON{} if a.cfg.Privacy.RecordOptinIP { if h := c.Request().Header.Get("X-Forwarded-For"); h != "" { meta["optin_ip"] = h } else if h := c.Request().RemoteAddr; h != "" { meta["optin_ip"] = strings.Split(h, ":")[0] } } // Confirm subscriptions in the DB. if err := a.core.ConfirmOptionSubscription(subUUID, req.ListUUIDs, meta); err != nil { a.log.Printf("error unsubscribing: %v", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.subConfirmedTitle"), "", a.i18n.Ts("public.subConfirmed"))) } var out optinTpl out.Lists = lists out.SubUUID = subUUID out.Title = a.i18n.T("public.confirmOptinSubTitle") return c.Render(http.StatusOK, "optin", out) } // SubscriptionFormPage handles subscription requests coming from public // HTML subscription forms. func (a *App) SubscriptionFormPage(c echo.Context) error { if !a.cfg.EnablePublicSubPage { return c.Render(http.StatusNotFound, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.invalidFeature"))) } // Get all public lists from the DB. lists, err := a.core.GetLists(models.ListTypePublic, models.ListStatusActive, true, nil) if err != nil { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorFetchingLists"))) } // There are no public lists available for subscription. if len(lists) == 0 { return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.noListsAvailable"))) } out := subFormTpl{} out.Title = a.i18n.T("public.sub") out.Lists = lists // Captcha configuration for template rendering. if a.cfg.Security.Captcha.Altcha.Enabled { out.Captcha.Enabled = true out.Captcha.Provider = "altcha" out.Captcha.Complexity = a.cfg.Security.Captcha.Altcha.Complexity } else if a.cfg.Security.Captcha.HCaptcha.Enabled { out.Captcha.Enabled = true out.Captcha.Provider = "hcaptcha" out.Captcha.Key = a.cfg.Security.Captcha.HCaptcha.Key } return c.Render(http.StatusOK, "subscription-form", out) } // SubscriptionForm handles subscription requests coming from public // HTML subscription forms. func (a *App) SubscriptionForm(c echo.Context) error { if !a.cfg.EnablePublicSubPage { return echo.NewHTTPError(http.StatusNotFound, a.i18n.T("public.invalidFeature")) } // If there's a nonce value, a bot could've filled the form. if c.FormValue("nonce") != "" { return echo.NewHTTPError(http.StatusBadGateway, a.i18n.T("public.invalidFeature")) } // Process CAPTCHA. if a.captcha.IsEnabled() { var val string // Get the appropriate captcha response field based on provider. switch a.captcha.GetProvider() { case captcha.ProviderHCaptcha: val = c.FormValue("h-captcha-response") case captcha.ProviderAltcha: val = c.FormValue("altcha") default: return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.invalidCaptcha"))) } if val == "" { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.invalidCaptcha"))) } err, ok := a.captcha.Verify(val) if err != nil { a.log.Printf("captcha request failed: %v", err) } if !ok { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.T("public.invalidCaptcha"))) } } hasOptin, err := a.processSubForm(c) if err != nil { e, ok := err.(*echo.HTTPError) if !ok { return err } return c.Render(e.Code, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", fmt.Sprintf("%s", e.Message))) } // If there were double optin lists, show the opt-in pending message instead of // the subscription confirmation message. msg := "public.subConfirmed" if hasOptin { msg = "public.subOptinPending" } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.subTitle"), "", a.i18n.Ts(msg))) } // PublicSubscription handles subscription requests coming from public // API calls. func (a *App) PublicSubscription(c echo.Context) error { if !a.cfg.EnablePublicSubPage { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.invalidFeature")) } hasOptin, err := a.processSubForm(c) if err != nil { return err } return c.JSON(http.StatusOK, okResp{struct { HasOptin bool `json:"has_optin"` }{hasOptin}}) } // LinkRedirect redirects a link UUID to its original underlying link // after recording the link click for a particular subscriber in the particular // campaign. These links are generated by {{ TrackLink }} tags in campaigns. func (a *App) LinkRedirect(c echo.Context) error { var ( linkUUID = c.Param("linkUUID") campUUID = c.Param("campUUID") ) // If tracking is globally disabled, resolve the URL without recording a click. if a.cfg.Privacy.DisableTracking { url, err := a.core.GetLinkURL(linkUUID) if err != nil { e := err.(*echo.HTTPError) return c.Render(e.Code, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", e.Error())) } return c.Redirect(http.StatusTemporaryRedirect, url) } // If individual tracking is disabled, do not record the subscriber ID. subUUID := c.Param("subUUID") if !a.cfg.Privacy.IndividualTracking { subUUID = "" } url, err := a.core.RegisterCampaignLinkClick(linkUUID, campUUID, subUUID) if err != nil { e := err.(*echo.HTTPError) return c.Render(e.Code, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", e.Error())) } return c.Redirect(http.StatusTemporaryRedirect, url) } // RegisterCampaignView registers a campaign view which comes in // the form of an pixel image request. Regardless of errors, this handler // should always render the pixel image bytes. The pixel URL is generated by // the {{ TrackView }} template tag in campaigns. func (a *App) RegisterCampaignView(c echo.Context) error { // If tracking is globally disabled, return the pixel without recording. if a.cfg.Privacy.DisableTracking { c.Response().Header().Set("Cache-Control", "no-cache") return c.Blob(http.StatusOK, "image/png", pixelPNG) } // If individual tracking is disabled, do not record the subscriber ID. subUUID := c.Param("subUUID") if !a.cfg.Privacy.IndividualTracking { subUUID = "" } // Exclude dummy hits from template previews. campUUID := c.Param("campUUID") if campUUID != dummyUUID && subUUID != dummyUUID { if err := a.core.RegisterCampaignView(campUUID, subUUID); err != nil { a.log.Printf("error registering campaign view: %s", err) } } c.Response().Header().Set("Cache-Control", "no-cache") return c.Blob(http.StatusOK, "image/png", pixelPNG) } // SelfExportSubscriberData pulls the subscriber's profile, list subscriptions, // campaign views and clicks and produces a JSON report that is then e-mailed // to the subscriber. This is a privacy feature and the data that's exported // is dependent on the configuration. func (a *App) SelfExportSubscriberData(c echo.Context) error { // Is export allowed? if !a.cfg.Privacy.AllowExport { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.invalidFeature"))) } // Get the subscriber's data. A single query that gets the profile, // list subscriptions, campaign views, and link clicks. Names of // private lists are replaced with "Private list". subUUID := c.Param("subUUID") data, b, err := a.exportSubscriberData(0, subUUID, a.cfg.Privacy.Exportable) if err != nil { a.log.Printf("error exporting subscriber data: %s", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } // Prepare the attachment e-mail. var msg bytes.Buffer if err := notifs.Tpls.ExecuteTemplate(&msg, notifs.TplSubscriberData, data); err != nil { a.log.Printf("error compiling notification template '%s': %v", notifs.TplSubscriberData, err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } // TODO: GetTplSubject should be moved to a utils package. subject, body := notifs.GetTplSubject(a.i18n.Ts("email.data.title"), msg.Bytes()) // E-mail the data as a JSON attachment to the subscriber. const fname = "data.json" if err := a.emailMsgr.Push(models.Message{ From: a.cfg.FromEmail, To: []string{data.Email}, Subject: subject, Body: body, Attachments: []models.Attachment{ { Name: fname, Content: b, Header: manager.MakeAttachmentHeader(fname, "base64", "application/json"), }, }, }); err != nil { a.log.Printf("error e-mailing subscriber profile: %s", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.dataSentTitle"), "", a.i18n.T("public.dataSent"))) } // WipeSubscriberData allows a subscriber to delete their data. The // profile and subscriptions are deleted, while the campaign_views and link // clicks remain as orphan data unconnected to any subscriber. func (a *App) WipeSubscriberData(c echo.Context) error { // Is wiping allowed? if !a.cfg.Privacy.AllowWipe { return c.Render(http.StatusBadRequest, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.invalidFeature"))) } subUUID := c.Param("subUUID") if err := a.core.DeleteSubscribers(nil, []string{subUUID}); err != nil { a.log.Printf("error wiping subscriber data: %s", err) return c.Render(http.StatusInternalServerError, tplMessage, makeMsgTpl(a.i18n.T("public.errorTitle"), "", a.i18n.Ts("public.errorProcessingRequest"))) } return c.Render(http.StatusOK, tplMessage, makeMsgTpl(a.i18n.T("public.dataRemovedTitle"), "", a.i18n.T("public.dataRemoved"))) } // AltchaChallenge generates a challenge for Altcha captcha. func (a *App) AltchaChallenge(c echo.Context) error { // Check if Altcha is enabled. if !a.captcha.IsEnabled() || a.captcha.GetProvider() != captcha.ProviderAltcha { return echo.NewHTTPError(http.StatusNotFound, "captcha not enabled") } // Generate challenge. out, err := a.captcha.GenerateChallenge() if err != nil { a.log.Printf("error generating altcha challenge: %v", err) return echo.NewHTTPError(http.StatusInternalServerError, "Error generating challenge") } // Return the challenge as JSON. c.Response().Header().Set("Content-Type", "application/json") return c.String(http.StatusOK, out) } // drawTransparentImage draws a transparent PNG of given dimensions // and returns the PNG bytes. func drawTransparentImage(h, w int) []byte { var ( img = image.NewRGBA(image.Rect(0, 0, w, h)) out = &bytes.Buffer{} ) _ = png.Encode(out, img) return out.Bytes() } // processSubForm processes an incoming form/public API subscription request. // The bool indicates whether there was subscription to an optin list so that // an appropriate message can be shown. func (a *App) processSubForm(c echo.Context) (bool, error) { // Get and validate fields. var req struct { Name string `form:"name" json:"name"` Email string `form:"email" json:"email"` FormListUUIDs []string `form:"l" json:"list_uuids"` } if err := c.Bind(&req); err != nil { return false, err } if len(req.FormListUUIDs) == 0 { return false, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.noListsSelected")) } // Validate fields. if len(req.Email) > 1000 { return false, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.invalidEmail")) } em, err := a.importer.SanitizeEmail(req.Email) if err != nil { return false, echo.NewHTTPError(http.StatusBadRequest, err.Error()) } req.Email = em req.Name = strings.TrimSpace(req.Name) if len(req.Name) == 0 { // If there's no name, use the name bit from the e-mail. req.Name = strings.Split(req.Email, "@")[0] } else if len(req.Name) > stdInputMaxLen { return false, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.invalidName")) } listUUIDs := pq.StringArray(req.FormListUUIDs) // Fetch the list types and ensure that they are not private. listTypes, err := a.core.GetListTypes(nil, req.FormListUUIDs) if err != nil { return false, echo.NewHTTPError(http.StatusInternalServerError, fmt.Sprintf("%s", err.(*echo.HTTPError).Message)) } for _, t := range listTypes { if t == models.ListTypePrivate { return false, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidUUID")) } } // Insert the subscriber into the DB. _, hasOptin, err := a.core.InsertSubscriber(models.Subscriber{ Name: req.Name, Email: req.Email, Status: models.SubscriberStatusEnabled, }, nil, listUUIDs, false, true) if err == nil { return hasOptin, nil } // Insert returned an error. Examine it. var lastErr = err // Subscriber already exists. Update subscriptions in the DB. if e, ok := err.(*echo.HTTPError); ok && e.Code == http.StatusConflict { // Get the subscriber from the DB by their email. sub, err := a.core.GetSubscriber(0, "", req.Email) if err != nil { return false, err } // Update the subscriber's subscriptions in the DB. _, hasOptin, err := a.core.UpdateSubscriberWithLists(sub.ID, sub, nil, listUUIDs, false, false, true, nil) if err == nil { return hasOptin, nil } lastErr = err } // Something else went wrong. if e, ok := lastErr.(*echo.HTTPError); ok { return false, echo.NewHTTPError(http.StatusBadRequest, fmt.Sprintf("%s", e.Message)) } return false, echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("public.errorProcessingRequest")) } ================================================ FILE: cmd/roles.go ================================================ package main import ( "fmt" "net/http" "strings" "github.com/knadh/listmonk/internal/auth" "github.com/labstack/echo/v4" ) // GetUserRoles retrieves roles. func (a *App) GetUserRoles(c echo.Context) error { // Get all roles. out, err := a.core.GetRoles() if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // GeListRoles retrieves roles. func (a *App) GeListRoles(c echo.Context) error { // Get all roles. out, err := a.core.GetListRoles() if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // CreateUserRole handles role creation. func (a *App) CreateUserRole(c echo.Context) error { var r auth.Role if err := c.Bind(&r); err != nil { return err } if err := a.validateUserRole(r); err != nil { return err } // Create the role in the DB. out, err := a.core.CreateRole(r) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // CreateListRole handles role creation. func (a *App) CreateListRole(c echo.Context) error { var r auth.ListRole if err := c.Bind(&r); err != nil { return err } if err := a.validateListRole(r); err != nil { return err } // Create the role in the DB. out, err := a.core.CreateListRole(r) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // UpdateUserRole handles role modification. func (a *App) UpdateUserRole(c echo.Context) error { id := getID(c) // ID 1 is reserved for the Super Admin user role. if id == auth.SuperAdminRoleID { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } // Incoming params. var r auth.Role if err := c.Bind(&r); err != nil { return err } if err := a.validateUserRole(r); err != nil { return err } // Validate. r.Name.String = strings.TrimSpace(r.Name.String) // Update the role in the DB. out, err := a.core.UpdateUserRole(id, r) if err != nil { return err } // Cache API tokens for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // UpdateListRole handles role modification. func (a *App) UpdateListRole(c echo.Context) error { // Get the role ID. id := getID(c) // ID 1 is reserved for the Super Admin user role. if id == auth.SuperAdminRoleID { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } // Incoming params. var r auth.ListRole if err := c.Bind(&r); err != nil { return err } if err := a.validateListRole(r); err != nil { return err } // Validate. r.Name.String = strings.TrimSpace(r.Name.String) // Update the role in the DB. out, err := a.core.UpdateListRole(id, r) if err != nil { return err } // Cache API tokens for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // DeleteRole handles (user|list) role deletion. func (a *App) DeleteRole(c echo.Context) error { // Get the role ID. id := getID(c) // ID 1 is reserved for the Super Admin user role. if id == auth.SuperAdminRoleID { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } // Delete the role from the DB. if err := a.core.DeleteRole(int(id)); err != nil { return err } // Cache API tokens for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } func (a *App) validateUserRole(r auth.Role) error { if !strHasLen(r.Name.String, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "name")) } for _, p := range r.Permissions { if _, ok := a.cfg.Permissions[p]; !ok { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", fmt.Sprintf("permission: %s", p))) } } return nil } func (a *App) validateListRole(r auth.ListRole) error { if !strHasLen(r.Name.String, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "name")) } for _, l := range r.Lists { for _, p := range l.Permissions { if p != auth.PermListGet && p != auth.PermListManage { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", fmt.Sprintf("list permission: %s", p))) } } } return nil } ================================================ FILE: cmd/settings.go ================================================ package main import ( "bytes" "encoding/json" "io" "net/http" "net/url" "regexp" "runtime" "strings" "syscall" "time" "unicode/utf8" "github.com/gdgvda/cron" "github.com/gofrs/uuid/v5" "github.com/jmoiron/sqlx/types" koanfjson "github.com/knadh/koanf/parsers/json" "github.com/knadh/koanf/providers/rawbytes" "github.com/knadh/koanf/v2" "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/internal/messenger/email" "github.com/knadh/listmonk/internal/notifs" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" ) const pwdMask = "•" type aboutHost struct { OS string `json:"os"` Machine string `json:"arch"` Hostname string `json:"hostname"` } type aboutSystem struct { NumCPU int `json:"num_cpu"` AllocMB uint64 `json:"memory_alloc_mb"` OSMB uint64 `json:"memory_from_os_mb"` } type about struct { Version string `json:"version"` Build string `json:"build"` GoVersion string `json:"go_version"` GoArch string `json:"go_arch"` Database types.JSONText `json:"database"` System aboutSystem `json:"system"` Host aboutHost `json:"host"` } var ( reAlphaNum = regexp.MustCompile(`[^a-z0-9\-]`) ) // GetSettings returns settings from the DB. func (a *App) GetSettings(c echo.Context) error { s, err := a.core.GetSettings() if err != nil { return err } // Empty out passwords. for i := range s.SMTP { s.SMTP[i].Password = strings.Repeat(pwdMask, utf8.RuneCountInString(s.SMTP[i].Password)) } for i := range s.BounceBoxes { s.BounceBoxes[i].Password = strings.Repeat(pwdMask, utf8.RuneCountInString(s.BounceBoxes[i].Password)) } for i := range s.Messengers { s.Messengers[i].Password = strings.Repeat(pwdMask, utf8.RuneCountInString(s.Messengers[i].Password)) } s.UploadS3AwsSecretAccessKey = strings.Repeat(pwdMask, utf8.RuneCountInString(s.UploadS3AwsSecretAccessKey)) s.SendgridKey = strings.Repeat(pwdMask, utf8.RuneCountInString(s.SendgridKey)) s.BouncePostmark.Password = strings.Repeat(pwdMask, utf8.RuneCountInString(s.BouncePostmark.Password)) s.BounceForwardEmail.Key = strings.Repeat(pwdMask, utf8.RuneCountInString(s.BounceForwardEmail.Key)) s.SecurityCaptcha.HCaptcha.Secret = strings.Repeat(pwdMask, utf8.RuneCountInString(s.SecurityCaptcha.HCaptcha.Secret)) s.OIDC.ClientSecret = strings.Repeat(pwdMask, utf8.RuneCountInString(s.OIDC.ClientSecret)) return c.JSON(http.StatusOK, okResp{s}) } // UpdateSettings returns settings from the DB. func (a *App) UpdateSettings(c echo.Context) error { // Unmarshal and marshal the fields once to sanitize the settings blob. var set models.Settings if err := c.Bind(&set); err != nil { return err } // Get the existing settings. cur, err := a.core.GetSettings() if err != nil { return err } // Validate and sanitize postback Messenger names along with SMTP names // (where each SMTP is also considered as a standalone messenger). // Duplicates are disallowed and "email" is a reserved name. names := map[string]bool{emailMsgr: true} // There should be at least one SMTP block that's enabled. has := false for i, s := range set.SMTP { if s.Enabled { has = true } // Sanitize and normalize the SMTP server name. name := reAlphaNum.ReplaceAllString(strings.ToLower(strings.TrimSpace(s.Name)), "-") if name != "" { if !strings.HasPrefix(name, "email-") { name = "email-" + name } if _, ok := names[name]; ok { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("settings.duplicateMessengerName", "name", name)) } names[name] = true } set.SMTP[i].Name = name // Assign a UUID. The frontend only sends a password when the user explicitly // changes the password. In other cases, the existing password in the DB // is copied while updating the settings and the UUID is used to match // the incoming array of SMTP blocks with the array in the DB. if s.UUID == "" { set.SMTP[i].UUID = uuid.Must(uuid.NewV4()).String() } // Ensure the HOST is trimmed of any whitespace. // This is a common mistake when copy-pasting SMTP settings. set.SMTP[i].Host = strings.TrimSpace(s.Host) // If there's no password coming in from the frontend, copy the existing // password by matching the UUID. if s.Password == "" { for _, c := range cur.SMTP { if s.UUID == c.UUID { set.SMTP[i].Password = c.Password } } } } if !has { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("settings.errorNoSMTP")) } // Always remove the trailing slash from the app root URL. set.AppRootURL = strings.TrimRight(set.AppRootURL, "/") // Bounce boxes. for i, s := range set.BounceBoxes { // Assign a UUID. The frontend only sends a password when the user explicitly // changes the password. In other cases, the existing password in the DB // is copied while updating the settings and the UUID is used to match // the incoming array of blocks with the array in the DB. if s.UUID == "" { set.BounceBoxes[i].UUID = uuid.Must(uuid.NewV4()).String() } // Ensure the HOST is trimmed of any whitespace. // This is a common mistake when copy-pasting SMTP settings. set.BounceBoxes[i].Host = strings.TrimSpace(s.Host) if d, _ := time.ParseDuration(s.ScanInterval); d.Minutes() < 1 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("settings.bounces.invalidScanInterval")) } // If there's no password coming in from the frontend, copy the existing // password by matching the UUID. if s.Password == "" { for _, c := range cur.BounceBoxes { if s.UUID == c.UUID { set.BounceBoxes[i].Password = c.Password } } } } for i, m := range set.Messengers { // UUID to keep track of password changes similar to the SMTP logic above. if m.UUID == "" { set.Messengers[i].UUID = uuid.Must(uuid.NewV4()).String() } if m.Password == "" { for _, c := range cur.Messengers { if m.UUID == c.UUID { set.Messengers[i].Password = c.Password } } } name := reAlphaNum.ReplaceAllString(strings.ToLower(m.Name), "") if _, ok := names[name]; ok { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("settings.duplicateMessengerName", "name", name)) } if len(name) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("settings.invalidMessengerName")) } set.Messengers[i].Name = name names[name] = true } // S3 password? if set.UploadS3AwsSecretAccessKey == "" { set.UploadS3AwsSecretAccessKey = cur.UploadS3AwsSecretAccessKey } if set.SendgridKey == "" { set.SendgridKey = cur.SendgridKey } if set.BouncePostmark.Password == "" { set.BouncePostmark.Password = cur.BouncePostmark.Password } if set.BounceForwardEmail.Key == "" { set.BounceForwardEmail.Key = cur.BounceForwardEmail.Key } if set.SecurityCaptcha.HCaptcha.Secret == "" { set.SecurityCaptcha.HCaptcha.Secret = cur.SecurityCaptcha.HCaptcha.Secret } if set.OIDC.ClientSecret == "" { set.OIDC.ClientSecret = cur.OIDC.ClientSecret } // OIDC user auto-creation is enabled. Validate. if set.OIDC.AutoCreateUsers { if set.OIDC.DefaultUserRoleID.Int < auth.SuperAdminRoleID { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", a.i18n.T("settings.security.OIDCDefaultRole"))) } } for n, v := range set.UploadExtensions { set.UploadExtensions[n] = strings.ToLower(strings.TrimPrefix(strings.TrimSpace(v), ".")) } // Domain blocklist / allowlist. doms := make([]string, 0, len(set.DomainBlocklist)) for _, d := range set.DomainBlocklist { if d = strings.TrimSpace(strings.ToLower(d)); d != "" { doms = append(doms, d) } } set.DomainBlocklist = doms doms = make([]string, 0, len(set.DomainAllowlist)) for _, d := range set.DomainAllowlist { if d = strings.TrimSpace(strings.ToLower(d)); d != "" { doms = append(doms, d) } } set.DomainAllowlist = doms // Validate and clean CORS domains. cors := make([]string, 0, len(set.SecurityCORSOrigins)) for _, d := range set.SecurityCORSOrigins { if d = strings.TrimSpace(d); d != "" { if d == "*" { cors = append(cors, d) continue } // Parse and validate the URL. u, err := url.Parse(d) if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidData")+": invalid CORS domain: "+d) } // Save clean scheme + host cors = append(cors, u.Scheme+"://"+u.Host) } } set.SecurityCORSOrigins = cors // Validate slow query caching cron. if set.CacheSlowQueries { if _, err := cron.ParseStandard(set.CacheSlowQueriesInterval); err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidData")+": slow query cron: "+err.Error()) } } // Update the settings in the DB. if err := a.core.UpdateSettings(set); err != nil { return err } return a.handleSettingsRestart(c) } // UpdateSettingsByKey updates a single setting key-value in the DB. func (a *App) UpdateSettingsByKey(c echo.Context) error { key := c.Param("key") if key == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidData")) } // Read the raw JSON body as the value. var b json.RawMessage if err := c.Bind(&b); err != nil { return err } // Update the value in the DB. if err := a.core.UpdateSettingsByKey(key, b); err != nil { return err } return a.handleSettingsRestart(c) } // handleSettingsRestart checks for running campaigns and either triggers an // immediate app restart or marks the app as needing a restart. func (a *App) handleSettingsRestart(c echo.Context) error { // If there are any active campaigns, don't do an auto reload and // warn the user on the frontend. if a.manager.HasRunningCampaigns() { a.Lock() a.needsRestart = true a.Unlock() return c.JSON(http.StatusOK, okResp{struct { NeedsRestart bool `json:"needs_restart"` }{true}}) } // No running campaigns. Reload the app. go func() { <-time.After(time.Millisecond * 500) a.chReload <- syscall.SIGHUP }() return c.JSON(http.StatusOK, okResp{true}) } // GetLogs returns the log entries stored in the log buffer. func (a *App) GetLogs(c echo.Context) error { return c.JSON(http.StatusOK, okResp{a.bufLog.Lines()}) } // TestSMTPSettings returns the log entries stored in the log buffer. func (a *App) TestSMTPSettings(c echo.Context) error { // Copy the raw JSON post body. reqBody, err := io.ReadAll(c.Request().Body) if err != nil { a.log.Printf("error reading SMTP test: %v", err) return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.internalError")) } // Load the JSON into koanf to parse SMTP settings properly including timestrings. ko := koanf.New(".") if err := ko.Load(rawbytes.Provider(reqBody), koanfjson.Parser()); err != nil { a.log.Printf("error unmarshalling SMTP test request: %v", err) return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.internalError")) } req := email.Server{} if err := ko.UnmarshalWithConf("", &req, koanf.UnmarshalConf{Tag: "json"}); err != nil { a.log.Printf("error scanning SMTP test request: %v", err) return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.internalError")) } to := ko.String("email") if to == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.missingFields", "name", "email")) } // Initialize a new SMTP pool. req.MaxConns = 1 req.IdleTimeout = time.Second * 2 req.PoolWaitTimeout = time.Second * 2 msgr, err := email.New("", req) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorCreating", "name", "SMTP", "error", err.Error())) } // Render the test email template body. var b bytes.Buffer if err := notifs.Tpls.ExecuteTemplate(&b, "smtp-test", nil); err != nil { a.log.Printf("error compiling notification template '%s': %v", "smtp-test", err) return err } m := models.Message{} m.From = a.cfg.FromEmail m.To = []string{to} m.Subject = a.i18n.T("settings.smtp.testConnection") m.Body = b.Bytes() if err := msgr.Push(m); err != nil { return echo.NewHTTPError(http.StatusInternalServerError, err.Error()) } return c.JSON(http.StatusOK, okResp{a.bufLog.Lines()}) } func (a *App) GetAboutInfo(c echo.Context) error { var mem runtime.MemStats runtime.ReadMemStats(&mem) out := a.about out.System.AllocMB = mem.Alloc / 1024 / 1024 out.System.OSMB = mem.Sys / 1024 / 1024 return c.JSON(http.StatusOK, out) } ================================================ FILE: cmd/subscribers.go ================================================ package main import ( "encoding/csv" "encoding/json" "fmt" "net/http" "net/textproto" "net/url" "strconv" "strings" "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/internal/i18n" "github.com/knadh/listmonk/internal/notifs" "github.com/knadh/listmonk/internal/subimporter" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" "github.com/lib/pq" ) const ( dummyUUID = "00000000-0000-0000-0000-000000000000" ) // subQueryReq is a "catch all" struct for reading various // subscriber related requests. type subQueryReq struct { Search string `json:"search"` Query string `json:"query"` ListIDs []int `json:"list_ids"` TargetListIDs []int `json:"target_list_ids"` SubscriberIDs []int `json:"ids"` Action string `json:"action"` Status string `json:"status"` SubscriptionStatus string `json:"subscription_status"` All bool `json:"all"` } // subOptin contains the data that's passed to the double opt-in e-mail template. type subOptin struct { models.Subscriber OptinURL string UnsubURL string Lists []models.List } var ( dummySubscriber = models.Subscriber{ Email: "demo@listmonk.app", Name: "Demo Subscriber", UUID: dummyUUID, Attribs: models.JSON{"city": "Bengaluru"}, } ) // GetSubscriber handles the retrieval of a single subscriber by ID. func (a *App) GetSubscriber(c echo.Context) error { user := auth.GetUser(c) // Check if the user has access to at least one of the lists on the subscriber. id := getID(c) if err := a.hasSubPerm(user, []int{id}); err != nil { return err } // Fetch the subscriber from the DB. out, err := a.core.GetSubscriber(id, "", "") if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // GetSubscriberActivity handles the retrieval of a subscriber's campaign views and link clicks. func (a *App) GetSubscriberActivity(c echo.Context) error { user := auth.GetUser(c) // Check if the user has access to at least one of the lists on the subscriber. id := getID(c) if err := a.hasSubPerm(user, []int{id}); err != nil { return err } // Fetch the subscriber activity from the DB. out, err := a.core.GetSubscriberActivity(id) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // QuerySubscribers handles querying subscribers based on an arbitrary SQL expression. func (a *App) QuerySubscribers(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Filter list IDs by permission. listIDs, err := a.filterListQueryByPerm("list_id", c.QueryParams(), user) if err != nil { return err } // Does the user have the subscribers:sql_query permission? query := formatSQLExp(c.FormValue("query")) if query != "" { if !user.HasPerm(auth.PermSubscribersSqlQuery) { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", auth.PermSubscribersSqlQuery)) } } var ( searchStr = strings.TrimSpace(c.FormValue("search")) subStatus = c.FormValue("subscription_status") order = c.FormValue("order") orderBy = c.FormValue("order_by") pg = a.pg.NewFromURL(c.Request().URL.Query()) ) // Query subscribers from the DB. res, total, err := a.core.QuerySubscribers(searchStr, query, listIDs, subStatus, order, orderBy, pg.Offset, pg.Limit) if err != nil { return err } out := models.PageResults{ Query: query, Search: searchStr, Results: res, Total: total, Page: pg.Page, PerPage: pg.PerPage, } return c.JSON(http.StatusOK, okResp{out}) } // ExportSubscribers handles querying subscribers based on an arbitrary SQL expression. func (a *App) ExportSubscribers(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Filter list IDs by permission. listIDs, err := a.filterListQueryByPerm("list_id", c.QueryParams(), user) if err != nil { return err } // Export only specific subscriber IDs? subIDs, err := getQueryInts("id", c.QueryParams()) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } // Filter by subscription status subStatus := c.QueryParam("subscription_status") // Does the user have the subscribers:sql_query permission? var ( searchStr = strings.TrimSpace(c.FormValue("search")) query = formatSQLExp(c.FormValue("query")) ) if query != "" { if !user.HasPerm(auth.PermSubscribersSqlQuery) { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", auth.PermSubscribersSqlQuery)) } } // Get the batched export iterator. exp, err := a.core.ExportSubscribers(searchStr, query, subIDs, listIDs, subStatus, a.cfg.DBBatchSize) if err != nil { return err } var ( hdr = c.Response().Header() wr = csv.NewWriter(c.Response()) ) hdr.Set(echo.HeaderContentType, echo.MIMEOctetStream) hdr.Set("Content-type", "text/csv") hdr.Set(echo.HeaderContentDisposition, "attachment; filename="+"subscribers.csv") hdr.Set("Content-Transfer-Encoding", "binary") hdr.Set("Cache-Control", "no-cache") wr.Write([]string{"uuid", "email", "name", "attributes", "status", "created_at", "updated_at"}) loop: // Iterate in batches until there are no more subscribers to export. for { out, err := exp() if err != nil { return err } if len(out) == 0 { break } for _, r := range out { if err = wr.Write([]string{r.UUID, r.Email, r.Name, r.Attribs, r.Status, r.CreatedAt.Time.String(), r.UpdatedAt.Time.String()}); err != nil { a.log.Printf("error streaming CSV export: %v", err) break loop } } // Flush CSV to stream after each batch. wr.Flush() } return nil } // CreateSubscriber handles the creation of a new subscriber. func (a *App) CreateSubscriber(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Get and validate fields. var req subimporter.SubReq if err := c.Bind(&req); err != nil { return err } // Validate fields. req, err := a.importer.ValidateFields(req) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, err.Error()) } // Filter lists against the current user's permitted lists. listIDs := user.FilterListsByPerm(auth.PermTypeManage, req.Lists) // Not a single permitted list? if len(req.Lists) > 0 && len(listIDs) == 0 { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", "lists")) } // Insert the subscriber into the DB. sub, _, err := a.core.InsertSubscriber(req.Subscriber, listIDs, nil, req.PreconfirmSubs, false) if err != nil { return err } return c.JSON(http.StatusOK, okResp{sub}) } // UpdateSubscriber handles modification of a subscriber. func (a *App) UpdateSubscriber(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Get and validate fields. req := struct { models.Subscriber Lists []int `json:"lists"` PreconfirmSubs bool `json:"preconfirm_subscriptions"` }{} if err := c.Bind(&req); err != nil { return err } // Sanitize and validate the email field. if em, err := a.importer.SanitizeEmail(req.Email); err != nil { return echo.NewHTTPError(http.StatusBadRequest, err.Error()) } else { req.Email = em } if req.Name != "" && !strHasLen(req.Name, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.invalidName")) } // Filter lists against the current user's permitted lists. listIDs := user.FilterListsByPerm(auth.PermTypeManage, req.Lists) // Not a single permitted list? if len(req.Lists) > 0 && len(listIDs) == 0 { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", "lists")) } // Update the subscriber in the DB. id := getID(c) // Get the user's permitted lists to pass to the update query so that lists on the subscribers // to which they don't have permissions are preserved/left as-is when deleteLists=true. allPerm, permittedLists := user.GetPermittedLists(auth.PermTypeManage) if allPerm { permittedLists = []int{} } out, _, err := a.core.UpdateSubscriberWithLists(id, req.Subscriber, listIDs, nil, req.PreconfirmSubs, true, false, permittedLists) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // SubscriberSendOptin sends an optin confirmation e-mail to a subscriber. func (a *App) SubscriberSendOptin(c echo.Context) error { // Fetch the subscriber. id := getID(c) out, err := a.core.GetSubscriber(id, "", "") if err != nil { return err } // Trigger the opt-in confirmation e-mail hook. if _, err := a.fnOptinNotify(out, nil); err != nil { return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.T("subscribers.errorSendingOptin")) } return c.JSON(http.StatusOK, okResp{true}) } // BlocklistSubscriber handles the blocklisting of a given subscriber. func (a *App) BlocklistSubscriber(c echo.Context) error { // Update the subscribers in the DB. id := getID(c) if err := a.core.BlocklistSubscribers([]int{id}); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // BlocklistSubscribers handles the blocklisting of one or more subscribers. func (a *App) BlocklistSubscribers(c echo.Context) error { var req subQueryReq if err := c.Bind(&req); err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error())) } if len(req.SubscriberIDs) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", "ids")) } // Update the subscribers in the DB. if err := a.core.BlocklistSubscribers(req.SubscriberIDs); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // ManageSubscriberLists handles bulk addition or removal of subscribers // from or to one or more target lists. // It takes either an ID in the URI, or a list of IDs in the request body. func (a *App) ManageSubscriberLists(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Is it an /:id call? var ( pID = c.Param("id") subIDs []int ) if pID != "" { id, _ := strconv.Atoi(pID) if id < 1 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } subIDs = append(subIDs, id) } var req subQueryReq if err := c.Bind(&req); err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error())) } if len(req.SubscriberIDs) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.errorNoIDs")) } if len(subIDs) == 0 { subIDs = req.SubscriberIDs } if len(req.TargetListIDs) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.errorNoListsGiven")) } // Filter lists against the current user's permitted lists. listIDs := user.FilterListsByPerm(auth.PermTypeGet|auth.PermTypeManage, req.TargetListIDs) // User doesn't have the required list permissions. if len(listIDs) == 0 { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", "lists")) } // Run the action in the DB. var err error switch req.Action { case "add": err = a.core.AddSubscriptions(subIDs, listIDs, req.Status) case "remove": err = a.core.DeleteSubscriptions(subIDs, listIDs) case "unsubscribe": err = a.core.UnsubscribeLists(subIDs, listIDs, nil) default: return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.invalidAction")) } if err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteSubscriber handles deletion of a single subscriber. func (a *App) DeleteSubscriber(c echo.Context) error { // Delete the subscribers from the DB. id := getID(c) if err := a.core.DeleteSubscribers([]int{id}, nil); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteSubscribers handles bulk deletion of one or more subscribers. func (a *App) DeleteSubscribers(c echo.Context) error { // Multiple IDs. ids, err := parseStringIDs(c.Request().URL.Query()["id"]) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error())) } if len(ids) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorInvalidIDs", "error", "ids")) } // Delete the subscribers from the DB. if err := a.core.DeleteSubscribers(ids, nil); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteSubscribersByQuery bulk deletes based on an // arbitrary SQL expression. func (a *App) DeleteSubscribersByQuery(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) var req subQueryReq if err := c.Bind(&req); err != nil { return err } req.Search = strings.TrimSpace(req.Search) req.Query = formatSQLExp(req.Query) if req.All { // If the "all" flag is set, ignore any subquery that may be present. req.Search = "" req.Query = "" } else if req.Search == "" && req.Query == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "query")) } // Does the user have the subscribers:sql_query permission? if req.Query != "" { if !user.HasPerm(auth.PermSubscribersSqlQuery) { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", auth.PermSubscribersSqlQuery)) } } // Delete the subscribers from the DB. if err := a.core.DeleteSubscribersByQuery(req.Search, req.Query, req.ListIDs, req.SubscriptionStatus); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // BlocklistSubscribersByQuery bulk blocklists subscribers // based on an arbitrary SQL expression. func (a *App) BlocklistSubscribersByQuery(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) var req subQueryReq if err := c.Bind(&req); err != nil { return err } req.Search = strings.TrimSpace(req.Search) req.Query = formatSQLExp(req.Query) if req.All { // If the "all" flag is set, ignore any subquery that may be present. req.Search = "" req.Query = "" } else if req.Search == "" && req.Query == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "query")) } // Does the user have the subscribers:sql_query permission? if req.Query != "" { if !user.HasPerm(auth.PermSubscribersSqlQuery) { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", auth.PermSubscribersSqlQuery)) } } // Update the subscribers in the DB. if err := a.core.BlocklistSubscribersByQuery(req.Search, req.Query, req.ListIDs, req.SubscriptionStatus); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // ManageSubscriberListsByQuery bulk adds/removes/unsubscribes subscribers // from one or more lists based on an arbitrary SQL expression. func (a *App) ManageSubscriberListsByQuery(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) var req subQueryReq if err := c.Bind(&req); err != nil { return err } if len(req.TargetListIDs) == 0 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.errorNoListsGiven")) } req.Search = strings.TrimSpace(req.Search) req.Query = formatSQLExp(req.Query) // Does the user have the subscribers:sql_query permission? if req.Query != "" { if !user.HasPerm(auth.PermSubscribersSqlQuery) { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", auth.PermSubscribersSqlQuery)) } } // Filter lists against the current user's permitted lists. sourceListIDs := user.FilterListsByPerm(auth.PermTypeGet|auth.PermTypeManage, req.ListIDs) targetListIDs := user.FilterListsByPerm(auth.PermTypeGet|auth.PermTypeManage, req.TargetListIDs) // Run the action in the DB. var err error switch req.Action { case "add": err = a.core.AddSubscriptionsByQuery(req.Search, req.Query, sourceListIDs, targetListIDs, req.Status, req.SubscriptionStatus) case "remove": err = a.core.DeleteSubscriptionsByQuery(req.Search, req.Query, sourceListIDs, targetListIDs, req.SubscriptionStatus) case "unsubscribe": err = a.core.UnsubscribeListsByQuery(req.Search, req.Query, sourceListIDs, targetListIDs, req.SubscriptionStatus) default: return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("subscribers.invalidAction")) } if err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteSubscriberBounces deletes all the bounces on a subscriber. func (a *App) DeleteSubscriberBounces(c echo.Context) error { // Delete the bounces from the DB. id := getID(c) if err := a.core.DeleteSubscriberBounces(id, ""); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // ExportSubscriberData pulls the subscriber's profile, // list subscriptions, campaign views and clicks and produces // a JSON report. This is a privacy feature and depends on the // configuration in a.Constants.Privacy. func (a *App) ExportSubscriberData(c echo.Context) error { // Get the subscriber's data. A single query that gets the profile, // list subscriptions, campaign views, and link clicks. Names of // private lists are replaced with "Private list". id := getID(c) _, b, err := a.exportSubscriberData(id, "", a.cfg.Privacy.Exportable) if err != nil { a.log.Printf("error exporting subscriber data: %s", err) return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("globals.messages.errorFetching", "name", "{globals.terms.subscribers}", "error", err.Error())) } // Set headers to force the browser to prompt for download. c.Response().Header().Set("Cache-Control", "no-cache") c.Response().Header().Set("Content-Disposition", `attachment; filename="data.json"`) return c.Blob(http.StatusOK, "application/json", b) } // exportSubscriberData collates the data of a subscriber including profile, // subscriptions, campaign_views, link_clicks (if they're enabled in the config) // and returns a formatted, indented JSON payload. Either takes a numeric id // and an empty subUUID or takes 0 and a string subUUID. func (a *App) exportSubscriberData(id int, subUUID string, exportables map[string]bool) (models.SubscriberExportProfile, []byte, error) { data, err := a.core.GetSubscriberProfileForExport(id, subUUID) if err != nil { return data, nil, err } // Filter out the non-exportable items. if _, ok := exportables["profile"]; !ok { data.Profile = nil } if _, ok := exportables["subscriptions"]; !ok { data.Subscriptions = nil } if _, ok := exportables["campaign_views"]; !ok { data.CampaignViews = nil } if _, ok := exportables["link_clicks"]; !ok { data.LinkClicks = nil } // Marshal the data into an indented payload. b, err := json.MarshalIndent(data, "", " ") if err != nil { a.log.Printf("error marshalling subscriber export data: %v", err) return data, nil, err } return data, b, nil } // hasSubPerm checks whether the current user has permission to access the given list // of subscriber IDs. func (a *App) hasSubPerm(u auth.User, subIDs []int) error { allPerm, listIDs := u.GetPermittedLists(auth.PermTypeGet | auth.PermTypeManage) // User has blanket get_all|manage_all permission. if allPerm { return nil } // Check whether the subscribers have the list IDs permitted to the user. res, err := a.core.HasSubscriberLists(subIDs, listIDs) if err != nil { return err } for id, has := range res { if !has { return echo.NewHTTPError(http.StatusForbidden, a.i18n.Ts("globals.messages.permissionDenied", "name", fmt.Sprintf("subscriber: %d", id))) } } return nil } // filterListQueryByPerm filters the list IDs in the query params and returns the list IDs to which the user has access. func (a *App) filterListQueryByPerm(param string, qp url.Values, user auth.User) ([]int, error) { var listIDs []int // If there are incoming list query params, filter them by permission. if qp.Has(param) { ids, err := getQueryInts(param, qp) if err != nil { return nil, echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } listIDs = user.FilterListsByPerm(auth.PermTypeGet|auth.PermTypeManage, ids) } // There are no incoming params. If the user doesn't have permission to get all subscribers, // filter by the lists they have access to. if len(listIDs) == 0 { if _, ok := user.PermissionsMap[auth.PermSubscribersGetAll]; !ok { if len(user.GetListIDs) > 0 { listIDs = user.GetListIDs } else { // User doesn't have access to any lists. listIDs = []int{-1} } } } return listIDs, nil } // formatSQLExp does basic sanitisation on arbitrary // SQL query expressions coming from the frontend. func formatSQLExp(q string) string { q = strings.TrimSpace(q) if len(q) == 0 { return "" } // Remove semicolon suffix. if q[len(q)-1] == ';' { q = q[:len(q)-1] } return q } // makeOptinNotifyHook returns an enclosed callback that sends optin confirmation e-mails. // This is plugged into the 'core' package to send optin confirmations when a new subscriber is // created via `core.CreateSubscriber()`. func makeOptinNotifyHook(unsubHeader bool, u *UrlConfig, q *models.Queries, i *i18n.I18n) func(sub models.Subscriber, listIDs []int) (int, error) { return func(sub models.Subscriber, listIDs []int) (int, error) { // Fetch double opt-in lists from the given list IDs. // Get the list of subscription lists where the subscriber hasn't confirmed. var lists = []models.List{} if err := q.GetSubscriberLists.Select(&lists, sub.ID, nil, pq.Array(listIDs), nil, models.SubscriptionStatusUnconfirmed, models.ListOptinDouble); err != nil { lo.Printf("error fetching lists for opt-in: %s", err) return 0, err } // None. if len(lists) == 0 { return 0, nil } var ( out = subOptin{Subscriber: sub, Lists: lists} qListIDs = url.Values{} ) // Construct the opt-in URL with list IDs. for _, l := range out.Lists { qListIDs.Add("l", l.UUID) } out.OptinURL = fmt.Sprintf(u.OptinURL, sub.UUID, qListIDs.Encode()) out.UnsubURL = fmt.Sprintf(u.UnsubURL, dummyUUID, sub.UUID) // Unsub headers. hdr := textproto.MIMEHeader{} hdr.Set(models.EmailHeaderSubscriberUUID, sub.UUID) // Attach List-Unsubscribe headers? if unsubHeader { unsubURL := fmt.Sprintf(u.UnsubURL, dummyUUID, sub.UUID) hdr.Set("List-Unsubscribe-Post", "List-Unsubscribe=One-Click") hdr.Set("List-Unsubscribe", `<`+unsubURL+`>`) } // Send the e-mail. if err := notifs.Notify([]string{sub.Email}, i.T("subscribers.optinSubject"), notifs.TplSubscriberOptin, out, hdr); err != nil { lo.Printf("error sending opt-in e-mail for subscriber %d (%s): %s", sub.ID, sub.UUID, err) return 0, err } return len(lists), nil } } ================================================ FILE: cmd/templates.go ================================================ package main import ( "errors" "html/template" "net/http" "regexp" "strconv" "strings" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" ) const ( // tplTag is the template tag that should be present in a template // as the placeholder for campaign bodies. tplTag = `{{ template "content" . }}` dummyTpl = `Hi there
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis et elit ac elit sollicitudin condimentum non a magna. Sed tempor mauris in facilisis vehicula. Aenean nisl urna, accumsan ac tincidunt vitae, interdum cursus massa. Interdum et malesuada fames ac ante ipsum primis in faucibus. Aliquam varius turpis et turpis lacinia placerat. Aenean id ligula a orci lacinia blandit at eu felis. Phasellus vel lobortis lacus. Suspendisse leo elit, luctus sed erat ut, venenatis fermentum ipsum. Donec bibendum neque quis.
Sub heading
Nam luctus dui non placerat mattis. Morbi non accumsan orci, vel interdum urna. Duis faucibus id nunc ut euismod. Curabitur et eros id erat feugiat fringilla in eget neque. Aliquam accumsan cursus eros sed faucibus.
Here is a link to listmonk.
` ) var ( regexpTplTag = regexp.MustCompile(`{{(\s+)?template\s+?"content"(\s+)?\.(\s+)?}}`) ) // GetTemplate handles the retrieval of a template func (a *App) GetTemplate(c echo.Context) error { // If no_body is true, blank out the body of the template from the response. noBody, _ := strconv.ParseBool(c.QueryParam("no_body")) // Get the template from the DB. id := getID(c) out, err := a.core.GetTemplate(id, noBody) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // GetTemplates handles retrieval of templates. func (a *App) GetTemplates(c echo.Context) error { // If no_body is true, blank out the body of the template from the response. noBody, _ := strconv.ParseBool(c.QueryParam("no_body")) // Fetch templates from the DB. out, err := a.core.GetTemplates("", noBody) if err != nil { return err } return c.JSON(http.StatusOK, okResp{out}) } // PreviewTemplate renders the HTML preview of a template in the DB. func (a *App) PreviewTemplate(c echo.Context) error { // Fetch one template from the DB. id := getID(c) tpl, err := a.core.GetTemplate(id, false) if err != nil { return err } // Render the template. out, err := a.previewTemplate(tpl) if err != nil { return err } return c.HTML(http.StatusOK, string(out)) } // PreviewTemplateBody renders the HTML preview of a template given its type and body. func (a *App) PreviewTemplateBody(c echo.Context) error { tpl := models.Template{ Type: c.FormValue("template_type"), Body: c.FormValue("body"), } // Body is posted with the request. if tpl.Type == "" { tpl.Type = models.TemplateTypeCampaign } if tpl.Type == models.TemplateTypeCampaign && !regexpTplTag.MatchString(tpl.Body) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("templates.placeholderHelp", "placeholder", tplTag)) } // Render the template. out, err := a.previewTemplate(tpl) if err != nil { return err } return c.HTML(http.StatusOK, string(out)) } // CreateTemplate handles template creation. func (a *App) CreateTemplate(c echo.Context) error { var o models.Template if err := c.Bind(&o); err != nil { return err } if err := a.validateTemplate(o); err != nil { return err } // Subject is only relevant for fixed tx templates. For campaigns, // the subject changes per campaign and is on models.Campaign. var funcs template.FuncMap if o.Type == models.TemplateTypeCampaign || o.Type == models.TemplateTypeCampaignVisual { o.Subject = "" funcs = a.manager.TemplateFuncs(nil) } else { funcs = a.manager.GenericTemplateFuncs() } // Compile the template and validate. if err := o.Compile(funcs); err != nil { return echo.NewHTTPError(http.StatusBadRequest, err.Error()) } // Create the template the in the DB. out, err := a.core.CreateTemplate(o.Name, o.Type, o.Subject, []byte(o.Body), o.BodySource) if err != nil { return err } // If it's a transactional template, cache it in the manager // to be used for arbitrary incoming tx message pushes. if o.Type == models.TemplateTypeTx { a.manager.CacheTpl(out.ID, &o) } return c.JSON(http.StatusOK, okResp{out}) } // UpdateTemplate handles template modification. func (a *App) UpdateTemplate(c echo.Context) error { var o models.Template if err := c.Bind(&o); err != nil { return err } if err := a.validateTemplate(o); err != nil { return err } // Subject is only relevant for fixed tx templates. For campaigns, // the subject changes per campaign and is on models.Campaign. var funcs template.FuncMap if o.Type == models.TemplateTypeCampaign || o.Type == models.TemplateTypeCampaignVisual { o.Subject = "" funcs = a.manager.TemplateFuncs(nil) } else { funcs = a.manager.GenericTemplateFuncs() } // Compile the template and validate. if err := o.Compile(funcs); err != nil { return echo.NewHTTPError(http.StatusBadRequest, err.Error()) } // Update the template in the DB. id := getID(c) out, err := a.core.UpdateTemplate(id, o.Name, o.Subject, []byte(o.Body), o.BodySource) if err != nil { return err } // If it's a transactional template, cache it. if out.Type == models.TemplateTypeTx { a.manager.CacheTpl(out.ID, &o) } return c.JSON(http.StatusOK, okResp{out}) } // TemplateSetDefault handles template modification. func (a *App) TemplateSetDefault(c echo.Context) error { // Update the template in the DB. id := getID(c) if err := a.core.SetDefaultTemplate(id); err != nil { return err } return a.GetTemplates(c) } // DeleteTemplate handles template deletion. func (a *App) DeleteTemplate(c echo.Context) error { // Delete the template from the DB. id := getID(c) if err := a.core.DeleteTemplate(id); err != nil { return err } // Delete cached in-memory template. a.manager.DeleteTpl(id) return c.JSON(http.StatusOK, okResp{true}) } // compileTemplate validates template fields. func (a *App) validateTemplate(o models.Template) error { if !strHasLen(o.Name, 1, stdInputMaxLen) { return errors.New(a.i18n.T("campaigns.fieldInvalidName")) } if o.Type == models.TemplateTypeCampaign && !regexpTplTag.MatchString(o.Body) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("templates.placeholderHelp", "placeholder", tplTag)) } if o.Type == models.TemplateTypeTx && strings.TrimSpace(o.Subject) == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.missingFields", "name", "subject")) } return nil } // previewTemplate renders the HTML preview of a template. func (a *App) previewTemplate(tpl models.Template) ([]byte, error) { var out []byte if tpl.Type == models.TemplateTypeCampaign || tpl.Type == models.TemplateTypeCampaignVisual { camp := models.Campaign{ UUID: dummyUUID, Name: a.i18n.T("templates.dummyName"), Subject: a.i18n.T("templates.dummySubject"), FromEmail: "dummy-campaign@listmonk.app", TemplateBody: tpl.Body, Body: dummyTpl, } if err := camp.CompileTemplate(a.manager.TemplateFuncs(&camp)); err != nil { return nil, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("templates.errorCompiling", "error", err.Error())) } // Render the message body. msg, err := a.manager.NewCampaignMessage(&camp, dummySubscriber) if err != nil { return nil, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("templates.errorRendering", "error", err.Error())) } out = msg.Body() } else { // Compile transactional template. if err := tpl.Compile(a.manager.GenericTemplateFuncs()); err != nil { return nil, echo.NewHTTPError(http.StatusBadRequest, err.Error()) } m := models.TxMessage{ Subject: tpl.Subject, } // Render the message. if err := m.Render(dummySubscriber, &tpl, a.manager.GenericTemplateFuncs()); err != nil { return nil, echo.NewHTTPError(http.StatusBadRequest, err.Error()) } out = m.Body } return out, nil } ================================================ FILE: cmd/tx.go ================================================ package main import ( "encoding/json" "fmt" "io" "net/http" "net/textproto" "strings" "github.com/knadh/listmonk/internal/manager" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" ) // SendTxMessage handles the sending of a transactional message. func (a *App) SendTxMessage(c echo.Context) error { var m models.TxMessage // If it's a multipart form, there may be file attachments. if strings.HasPrefix(c.Request().Header.Get("Content-Type"), "multipart/form-data") { form, err := c.MultipartForm() if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", err.Error())) } data, ok := form.Value["data"] if !ok || len(data) != 1 { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "data")) } // Parse the JSON data. if err := json.Unmarshal([]byte(data[0]), &m); err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", fmt.Sprintf("data: %s", err.Error()))) } // Attach files. for _, f := range form.File["file"] { file, err := f.Open() if err != nil { return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("globals.messages.invalidFields", "name", fmt.Sprintf("file: %s", err.Error()))) } defer file.Close() b, err := io.ReadAll(file) if err != nil { return echo.NewHTTPError(http.StatusInternalServerError, a.i18n.Ts("globals.messages.invalidFields", "name", fmt.Sprintf("file: %s", err.Error()))) } m.Attachments = append(m.Attachments, models.Attachment{ Name: f.Filename, Header: manager.MakeAttachmentHeader(f.Filename, "base64", f.Header.Get("Content-Type")), Content: b, }) } } else if err := c.Bind(&m); err != nil { return err } // Validate fields. if r, err := a.validateTxMessage(m); err != nil { return err } else { m = r } // Get the cached tx template. tpl, err := a.manager.GetTpl(m.TemplateID) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.notFound", "name", fmt.Sprintf("template %d", m.TemplateID))) } var ( num = len(m.SubscriberEmails) isEmails = true ) if len(m.SubscriberIDs) > 0 { num = len(m.SubscriberIDs) isEmails = false } notFound := []string{} for n := range num { var sub models.Subscriber if m.SubscriberMode == models.TxSubModeExternal { // `external`: Always create an ephemeral "subscriber" and don't // lookup in the DB. sub = models.Subscriber{ Email: m.SubscriberEmails[n], } } else { // Default/fallback mode: lookup subscriber in DB. var ( subID int subEmail string ) if !isEmails { subID = m.SubscriberIDs[n] } else { subEmail = m.SubscriberEmails[n] } var err error sub, err = a.core.GetSubscriber(subID, "", subEmail) if err != nil { if er, ok := err.(*echo.HTTPError); ok && er.Code == http.StatusBadRequest { // `fallback`: Create an ephemeral "subscriber" if the subscriber wasn't found. if m.SubscriberMode == models.TxSubModeFallback { sub = models.Subscriber{ Email: subEmail, } } else { // `default`: log error and continue. notFound = append(notFound, fmt.Sprintf("%v", er.Message)) continue } } else { return err } } } // Render the message. if err := m.Render(sub, tpl, a.manager.GenericTemplateFuncs()); err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.errorFetching", "name")) } // Prepare the final message. msg := models.Message{} msg.Subscriber = sub msg.To = []string{sub.Email} msg.From = m.FromEmail msg.Subject = m.Subject msg.ContentType = m.ContentType msg.Messenger = m.Messenger msg.Body = m.Body msg.AltBody = []byte(m.AltBody) for _, a := range m.Attachments { msg.Attachments = append(msg.Attachments, models.Attachment{ Name: a.Name, Header: a.Header, Content: a.Content, }) } // Optional headers. if len(m.Headers) != 0 { msg.Headers = make(textproto.MIMEHeader, len(m.Headers)) for _, set := range m.Headers { for hdr, val := range set { msg.Headers.Add(hdr, val) } } } if err := a.manager.PushMessage(msg); err != nil { a.log.Printf("error sending message (%s): %v", msg.Subject, err) return err } } if len(notFound) > 0 { return echo.NewHTTPError(http.StatusBadRequest, strings.Join(notFound, "; ")) } return c.JSON(http.StatusOK, okResp{true}) } // validateTxMessage validates the tx message fields. func (a *App) validateTxMessage(m models.TxMessage) (models.TxMessage, error) { if len(m.SubscriberEmails) > 0 && m.SubscriberEmail != "" { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "do not send `subscriber_email`")) } if len(m.SubscriberIDs) > 0 && m.SubscriberID != 0 { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "do not send `subscriber_id`")) } if m.SubscriberEmail != "" { m.SubscriberEmails = append(m.SubscriberEmails, m.SubscriberEmail) } if m.SubscriberID != 0 { m.SubscriberIDs = append(m.SubscriberIDs, m.SubscriberID) } // Validate subscriber_mode. if m.SubscriberMode == "" { m.SubscriberMode = models.TxSubModeDefault } switch m.SubscriberMode { case models.TxSubModeDefault: // Need subscriber_emails OR subscriber_ids, but not both. if (len(m.SubscriberEmails) == 0 && len(m.SubscriberIDs) == 0) || (len(m.SubscriberEmails) > 0 && len(m.SubscriberIDs) > 0) { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "send subscriber_emails OR subscriber_ids")) } case models.TxSubModeFallback, models.TxSubModeExternal: // `fallback` and `external` can only use subscriber_emails. if len(m.SubscriberIDs) > 0 { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "subscriber_ids not allowed in fallback or external mode")) } if len(m.SubscriberEmails) == 0 { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "subscriber_emails")) } default: return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "subscriber_mode")) } for n, email := range m.SubscriberEmails { if email != "" { em, err := a.importer.SanitizeEmail(email) if err != nil { return m, echo.NewHTTPError(http.StatusBadRequest, err.Error()) } m.SubscriberEmails[n] = em } } if m.FromEmail == "" { m.FromEmail = a.cfg.FromEmail } if m.Messenger == "" { m.Messenger = emailMsgr } else if !a.manager.HasMessenger(m.Messenger) { return m, echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("campaigns.fieldInvalidMessenger", "name", m.Messenger)) } return m, nil } ================================================ FILE: cmd/updates.go ================================================ package main import ( "encoding/json" "io" "net/http" "regexp" "time" "golang.org/x/mod/semver" ) const updateCheckURL = "https://update.listmonk.app/update.json" type AppUpdate struct { Update struct { ReleaseVersion string `json:"release_version"` ReleaseDate string `json:"release_date"` URL string `json:"url"` Description string `json:"description"` // This is computed and set locally based on the local version. IsNew bool `json:"is_new"` } `json:"update"` Messages []struct { Date string `json:"date"` Title string `json:"title"` Description string `json:"description"` URL string `json:"url"` Priority string `json:"priority"` } `json:"messages"` } var reSemver = regexp.MustCompile(`-(.*)`) // checkUpdates is a blocking function that checks for updates to the app // at the given intervals. On detecting a new update (new semver), it // sets the global update status that renders a prompt on the UI. func (a *App) checkUpdates(curVersion string, interval time.Duration) { // Strip -* suffix. curVersion = reSemver.ReplaceAllString(curVersion, "") fnCheck := func() { resp, err := http.Get(updateCheckURL) if err != nil { a.log.Printf("error checking for remote update: %v", err) return } if resp.StatusCode != 200 { a.log.Printf("non 200 response on remote update check: %d", resp.StatusCode) return } b, err := io.ReadAll(resp.Body) if err != nil { a.log.Printf("error reading remote update payload: %v", err) return } resp.Body.Close() var out AppUpdate if err := json.Unmarshal(b, &out); err != nil { a.log.Printf("error unmarshalling remote update payload: %v", err) return } // There is an update. Set it on the global app state. if semver.IsValid(out.Update.ReleaseVersion) { v := reSemver.ReplaceAllString(out.Update.ReleaseVersion, "") if semver.Compare(v, curVersion) > 0 { out.Update.IsNew = true a.log.Printf("new update %s found", out.Update.ReleaseVersion) } } a.Lock() a.update = &out a.Unlock() } // Give a 15 minute buffer after app start in case the admin wants to disable // update checks entirely and not make a request to upstream. time.Sleep(time.Minute * 15) fnCheck() // Thereafter, check every $interval. ticker := time.NewTicker(interval) defer ticker.Stop() for range ticker.C { fnCheck() } } ================================================ FILE: cmd/upgrade.go ================================================ package main import ( "fmt" "log" "strings" "github.com/jmoiron/sqlx" "github.com/knadh/koanf/v2" "github.com/knadh/listmonk/internal/migrations" "github.com/knadh/stuffbin" "github.com/lib/pq" "golang.org/x/mod/semver" ) // migFunc represents a migration function for a particular version. // fn (generally) executes database migrations and additionally // takes the filesystem and config objects in case there are additional bits // of logic to be performed before executing upgrades. fn is idempotent. type migFunc struct { version string fn func(*sqlx.DB, stuffbin.FileSystem, *koanf.Koanf, *log.Logger) error } // migList is the list of available migList ordered by the semver. // Each migration is a Go file in internal/migrations named after the semver. // The functions are named as: v0.7.0 => migrations.V0_7_0() and are idempotent. var migList = []migFunc{ {"v0.4.0", migrations.V0_4_0}, {"v0.7.0", migrations.V0_7_0}, {"v0.8.0", migrations.V0_8_0}, {"v0.9.0", migrations.V0_9_0}, {"v1.0.0", migrations.V1_0_0}, {"v2.0.0", migrations.V2_0_0}, {"v2.1.0", migrations.V2_1_0}, {"v2.2.0", migrations.V2_2_0}, {"v2.3.0", migrations.V2_3_0}, {"v2.4.0", migrations.V2_4_0}, {"v2.5.0", migrations.V2_5_0}, {"v3.0.0", migrations.V3_0_0}, {"v4.0.0", migrations.V4_0_0}, {"v4.1.0", migrations.V4_1_0}, {"v5.0.0", migrations.V5_0_0}, {"v5.1.0", migrations.V5_1_0}, {"v6.0.0", migrations.V6_0_0}, {"v6.1.0", migrations.V6_1_0}, } // upgrade upgrades the database to the current version by running SQL migration files // for all version from the last known version to the current one. // If record is false, migration versions are not recorded in the DB (used for nightly builds). func upgrade(db *sqlx.DB, fs stuffbin.FileSystem, prompt bool, record bool) { if prompt { var ok string fmt.Printf("** IMPORTANT: Take a backup of the database before upgrading.\n") fmt.Print("continue (y/n)? ") if _, err := fmt.Scanf("%s", &ok); err != nil { lo.Fatalf("error reading value from terminal: %v", err) } if strings.ToLower(ok) != "y" { fmt.Println("upgrade cancelled") return } } _, toRun, err := getPendingMigrations(db) if err != nil { lo.Fatalf("error checking migrations: %v", err) } // No migrations to run. if len(toRun) == 0 { lo.Printf("no upgrades to run. Database is up to date.") return } // Execute migrations in succession. for _, m := range toRun { lo.Printf("running migration %s", m.version) if err := m.fn(db, fs, ko, lo); err != nil { lo.Fatalf("error running migration %s: %v", m.version, err) } // Record the migration version in the settings table. There was no // settings table until v0.7.0, so ignore the no-table errors. // For nightly builds, skip recording so migrations re-run on each boot. if record { if err := recordMigrationVersion(m.version, db); err != nil { if isTableNotExistErr(err) { continue } lo.Fatalf("error recording migration version %s: %v", m.version, err) } } } lo.Printf("upgrade complete") } // checkUpgrade checks if the current database schema matches the expected // binary version. func checkUpgrade(db *sqlx.DB) { lastVer, toRun, err := getPendingMigrations(db) if err != nil { lo.Fatalf("error checking migrations: %v", err) } // No migrations to run. if len(toRun) == 0 { return } var vers []string for _, m := range toRun { vers = append(vers, m.version) } lo.Fatalf(`there are %d pending database upgrade(s): %v. The last upgrade was %s. Backup the database and run listmonk --upgrade`, len(toRun), vers, lastVer) } // getPendingMigrations gets the pending migrations by comparing the last // recorded migration in the DB against all migrations listed in `migrations`. func getPendingMigrations(db *sqlx.DB) (string, []migFunc, error) { lastVer, err := getLastMigrationVersion(db) if err != nil { return "", nil, err } // Iterate through the migration versions and get everything above the last // upgraded semver. var toRun []migFunc for i, m := range migList { if semver.Compare(m.version, lastVer) > 0 { toRun = migList[i:] break } } return lastVer, toRun, nil } // getLastMigrationVersion returns the last migration semver recorded in the DB. // If there isn't any, `v0.0.0` is returned. func getLastMigrationVersion(db *sqlx.DB) (string, error) { var v string if err := db.Get(&v, ` SELECT COALESCE( (SELECT value->>-1 FROM settings WHERE key='migrations'), 'v0.0.0')`); err != nil { if isTableNotExistErr(err) { return "v0.0.0", nil } return v, err } return v, nil } // isTableNotExistErr checks if the given error represents a Postgres/pq // "table does not exist" error. func isTableNotExistErr(err error) bool { if p, ok := err.(*pq.Error); ok { // `settings` table does not exist. It was introduced in v0.7.0. if p.Code == "42P01" { return true } } return false } ================================================ FILE: cmd/users.go ================================================ package main import ( "net/http" "regexp" "strings" "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/internal/core" "github.com/knadh/listmonk/internal/utils" "github.com/knadh/listmonk/models" "github.com/labstack/echo/v4" "github.com/pquerna/otp/totp" "gopkg.in/volatiletech/null.v6" ) var ( reUsername = regexp.MustCompile(`^[a-zA-Z0-9_\-\.@]+$`) ) // GetUser retrieves a single user by ID. func (a *App) GetUser(c echo.Context) error { // Get the user from the DB. id := getID(c) out, err := a.core.GetUser(id, "", "") if err != nil { return err } // Blank out the password hash in the response. out.Password = null.String{} return c.JSON(http.StatusOK, okResp{out}) } // GetUsers retrieves all users. func (a *App) GetUsers(c echo.Context) error { // Get all users from the DB. out, err := a.core.GetUsers() if err != nil { return err } // Blank out the password hash in the response. for n := range out { out[n].Password = null.String{} } return c.JSON(http.StatusOK, okResp{out}) } // CreateUser handles user creation. func (a *App) CreateUser(c echo.Context) error { var u auth.User if err := c.Bind(&u); err != nil { return err } u.Username = strings.TrimSpace(u.Username) u.Name = strings.TrimSpace(u.Name) email := strings.ToLower(strings.TrimSpace(u.Email.String)) // Validate fields. if !strHasLen(u.Username, 3, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username")) } if !reUsername.MatchString(u.Username) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username")) } if u.Type != auth.UserTypeAPI { if !utils.ValidateEmail(email) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "email")) } if u.PasswordLogin { if !strHasLen(u.Password.String, 8, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } } u.Email = null.String{String: email, Valid: true} } if u.Name == "" { u.Name = u.Username } // Create the user in the DB. user, err := a.core.CreateUser(u) if err != nil { return err } // Blank out the password hash in the response. if user.Type != auth.UserTypeAPI { user.Password = null.String{} } // Cache the API token for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{user}) } // UpdateUser handles user modification. func (a *App) UpdateUser(c echo.Context) error { // Incoming params. var u auth.User if err := c.Bind(&u); err != nil { return err } u.Username = strings.TrimSpace(u.Username) u.Name = strings.TrimSpace(u.Name) email := strings.ToLower(strings.TrimSpace(u.Email.String)) // Validate fields. if !strHasLen(u.Username, 3, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username")) } if !reUsername.MatchString(u.Username) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "username")) } // Get the user ID. id := getID(c) if u.Type != auth.UserTypeAPI { if !utils.ValidateEmail(email) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "email")) } // Validate password if password login is enabled. if u.PasswordLogin && u.Password.String != "" { if !strHasLen(u.Password.String, 8, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } if u.Password.String != "" { // If a password is sent, validate it before updating in the DB. If it's not set, leave the password in the DB untouched. if !strHasLen(u.Password.String, 8, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } } else { // Get the user from the DB. user, err := a.core.GetUser(id, "", "") if err != nil { return err } // If password login is enabled, but there's no password in the DB and there's no incoming // password, throw an error. if !user.HasPassword { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } } } u.Email = null.String{String: email, Valid: true} } // Default the name to username if not set. if u.Name == "" { u.Name = u.Username } // Update the user in the DB. user, err := a.core.UpdateUser(id, u) if err != nil { return err } // Blank out the password hash in the response. user.Password = null.String{} // Cache the API token for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{user}) } // DeleteUser handles the deletion of a single user by ID. func (a *App) DeleteUser(c echo.Context) error { // Delete the user(s) from the DB. id := getID(c) if err := a.core.DeleteUsers([]int{id}); err != nil { return err } // Cache the API token for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DeleteUsers handles user deletion, either a single one (ID in the URI), or a list. func (a *App) DeleteUsers(c echo.Context) error { ids, err := getQueryInts("id", c.QueryParams()) if err != nil { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidID")) } // Delete the user(s) from the DB. if err := a.core.DeleteUsers(ids); err != nil { return err } // Cache the API token for in-memory, off-DB /api/* request auth. if _, err := cacheUsers(a.core, a.auth); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // GetUserProfile fetches the uesr profile for the currently logged in user. func (a *App) GetUserProfile(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Blank out the password hash in the response. user.Password.String = "" user.Password.Valid = false return c.JSON(http.StatusOK, okResp{user}) } // UpdateUserProfile update's the current user's profile. func (a *App) UpdateUserProfile(c echo.Context) error { // Get the authenticated user. user := auth.GetUser(c) // Incoming params. u := auth.User{} if err := c.Bind(&u); err != nil { return err } u.PasswordLogin = user.PasswordLogin u.Name = strings.TrimSpace(u.Name) email := strings.TrimSpace(u.Email.String) // Validate fields. if user.PasswordLogin { if !utils.ValidateEmail(email) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "email")) } u.Email = null.String{String: email, Valid: true} } if u.PasswordLogin && u.Password.String != "" { if !strHasLen(u.Password.String, 8, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } } // Update the user in the DB. out, err := a.core.UpdateUserProfile(user.ID, u) if err != nil { return err } // Blank out the password hash in the response. out.Password = null.String{} return c.JSON(http.StatusOK, okResp{out}) } // EnableTOTP enables TOTP 2FA for a user after verifying the code. func (a *App) EnableTOTP(c echo.Context) error { var ( u = c.Get(auth.UserHTTPCtxKey).(auth.User) secret = strings.TrimSpace(c.FormValue("secret")) code = strings.TrimSpace(c.FormValue("code")) ) if secret == "" || code == "" { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("globals.messages.invalidFields")) } // If password login is disabled, can't enable TOTP. if !u.PasswordLogin { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("public.invalidFeature")) } // If TOTP is already enabled, don't allow re-enabling. if u.TwofaType == models.TwofaTypeTOTP { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("users.twoFAAlreadyEnabled")) } // Verify the TOTP code. valid := totp.Validate(code, secret) if !valid { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("users.invalidTOTPCode")) } // Enable TOTP in the DB. if err := a.core.SetTwoFA(u.ID, models.TwofaTypeTOTP, secret); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // DisableTOTP disables TOTP 2FA for a user after verifying the password. func (a *App) DisableTOTP(c echo.Context) error { var ( u = c.Get(auth.UserHTTPCtxKey).(auth.User) password = c.FormValue("password") ) // TOTP isn't enabled. if u.TwofaType != models.TwofaTypeTOTP { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.T("users.twoFANotEnabled")) } // Validate password. if !strHasLen(password, 8, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, a.i18n.Ts("globals.messages.invalidFields", "name", "password")) } // Verify the password. if _, err := a.core.LoginUser(u.Username, password); err != nil { return echo.NewHTTPError(http.StatusForbidden, a.i18n.T("users.invalidPassword")) } // Disable TOTP in the DB. if err := a.core.SetTwoFA(u.ID, models.TwofaTypeNone, ""); err != nil { return err } return c.JSON(http.StatusOK, okResp{true}) } // cacheUsers fetches (API) users and caches them in the auth module. // It also returns a bool indicating whether there are any actual users in the DB at all, // which if there aren't, the first time user setup needs to be run. func cacheUsers(co *core.Core, a *auth.Auth) (bool, error) { users, err := co.GetUsers() if err != nil { return false, err } hasUser := false apiUsers := make([]auth.User, 0, len(users)) for _, u := range users { if u.Type == auth.UserTypeAPI && u.Status == auth.UserStatusEnabled { apiUsers = append(apiUsers, u) } if u.Type == auth.UserTypeUser { hasUser = true } } a.CacheAPIUsers(apiUsers) return hasUser, nil } ================================================ FILE: cmd/utils.go ================================================ package main import ( "crypto/rand" "fmt" "net/url" "path/filepath" "regexp" "slices" "strconv" "strings" ) var ( regexpSpaces = regexp.MustCompile(`[\s]+`) ) // inArray checks if a string is present in a list of strings. func inArray(val string, vals []string) (ok bool) { return slices.Contains(vals, val) } // makeFilename sanitizes a filename (user supplied upload filenames). func makeFilename(fName string) string { name := strings.TrimSpace(fName) if name == "" { name, _ = generateRandomString(10) } // replace whitespace with "-" name = regexpSpaces.ReplaceAllString(name, "-") return filepath.Base(name) } // appendSuffixToFilename adds a string suffix to the filename while keeping the file extension. func appendSuffixToFilename(filename, suffix string) string { ext := filepath.Ext(filename) name := strings.TrimSuffix(filename, ext) return fmt.Sprintf("%s_%s%s", name, suffix, ext) } // makeMsgTpl takes a page title, heading, and message and returns // a msgTpl that can be rendered as an HTML view. This is used for // rendering arbitrary HTML views with error and success messages. func makeMsgTpl(pageTitle, heading, msg string) msgTpl { if heading == "" { heading = pageTitle } err := msgTpl{} err.Title = pageTitle err.MessageTitle = heading err.Message = msg return err } // parseStringIDs takes a slice of numeric string IDs and // parses each number into an int64 and returns a slice of the // resultant values. func parseStringIDs(s []string) ([]int, error) { vals := make([]int, 0, len(s)) for _, v := range s { i, err := strconv.Atoi(v) if err != nil { return nil, err } if i < 1 { return nil, fmt.Errorf("%d is not a valid ID", i) } vals = append(vals, i) } return vals, nil } // generateRandomString generates a cryptographically random, alphanumeric string of length n. func generateRandomString(n int) (string, error) { const dictionary = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" var bytes = make([]byte, n) if _, err := rand.Read(bytes); err != nil { return "", err } for k, v := range bytes { bytes[k] = dictionary[v%byte(len(dictionary))] } return string(bytes), nil } // strHasLen checks if the given string has a length within min-max. func strHasLen(str string, min, max int) bool { return len(str) >= min && len(str) <= max } // getQueryInts parses the list of given query param values into ints. func getQueryInts(param string, qp url.Values) ([]int, error) { var out []int if vals, ok := qp[param]; ok { for _, v := range vals { if v == "" { continue } listID, err := strconv.Atoi(v) if err != nil { return nil, err } out = append(out, listID) } } return out, nil } ================================================ FILE: config.toml.sample ================================================ [app] # Interface and port where the app will run its webserver. The default value # of localhost will only listen to connections from the current machine. To # listen on all interfaces use '0.0.0.0'. To listen on the default web address # port, use port 80 (this will require running with elevated permissions). address = "localhost:9000" # Database. [db] host = "localhost" port = 5432 user = "listmonk" password = "listmonk" # Ensure that this database has been created in Postgres. database = "listmonk" ssl_mode = "disable" max_open = 25 max_idle = 25 max_lifetime = "300s" # Optional space separated Postgres DSN params. eg: "application_name=listmonk gssencmode=disable" params = "" ================================================ FILE: dev/.gitignore ================================================ !config.toml ================================================ FILE: dev/README.md ================================================ # Docker suite for development **NOTE**: This exists only for local development. If you're interested in using Docker for a production setup, visit the [docs](https://listmonk.app/docs/installation/#docker) instead. ### Objective The purpose of this Docker suite for local development is to isolate all the dev dependencies in a Docker environment. The containers have a host volume mounted inside for the entire app directory. This helps us to not do a full `docker build` for every single local change, only restarting the Docker environment is enough. ## Setting up a dev suite To spin up a local suite of: - PostgreSQL - Mailhog - Node.js frontend app - Golang backend app ### Verify your config file The config file provided at `dev/config.toml` will be used when running the containerized development stack. Make sure the values set within are suitable for the feature you're trying to develop. ### Setup DB Running this will build the appropriate images and initialize the database. ```bash make init-dev-docker ``` ### Start frontend and backend apps Running this start your local development stack. ```bash make dev-docker ``` Visit `http://localhost:8080` on your browser. ### Tear down This will tear down all the data, including DB. ```bash make rm-dev-docker ``` ### See local changes in action - Backend: Anytime you do a change to the Go app, it needs to be compiled. Just run `make dev-docker` again and that should automatically handle it for you. - Frontend: Anytime you change the frontend code, you don't need to do anything. Since `yarn` is watching for all the changes and we have mounted the code inside the docker container, `yarn` server automatically restarts. ================================================ FILE: dev/app.Dockerfile ================================================ FROM golang:1.24.1 AS go FROM node:16 AS node COPY --from=go /usr/local/go /usr/local/go ENV GOPATH /go ENV CGO_ENABLED=0 ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH WORKDIR /app CMD [ "sleep infinity" ] ================================================ FILE: dev/config.toml ================================================ [app] # Interface and port where the app will run its webserver. The default value # of localhost will only listen to connections from the current machine. To # listen on all interfaces use '0.0.0.0'. To listen on the default web address # port, use port 80 (this will require running with elevated permissions). address = "0.0.0.0:9000" # Database. [db] host = "db" port = 5432 user = "listmonk-dev" password = "listmonk-dev" # Ensure that this database has been created in Postgres. database = "listmonk-dev" ssl_mode = "disable" max_open = 25 max_idle = 25 max_lifetime = "300s" # Optional space separated Postgres DSN params. eg: "application_name=listmonk gssencmode=disable" params = "" ================================================ FILE: dev/docker-compose.yml ================================================ version: "3" services: adminer: image: adminer:4.8.1-standalone restart: always ports: - 8070:8080 networks: - listmonk-dev mailhog: image: mailhog/mailhog:v1.0.1 ports: - "1025:1025" # SMTP - "8025:8025" # UI networks: - listmonk-dev db: image: postgres:13 ports: - "5432:5432" networks: - listmonk-dev environment: - POSTGRES_PASSWORD=listmonk-dev - POSTGRES_USER=listmonk-dev - POSTGRES_DB=listmonk-dev restart: unless-stopped volumes: - type: volume source: listmonk-dev-db target: /var/lib/postgresql/data front: build: context: ../ dockerfile: dev/app.Dockerfile command: ["make", "run-frontend"] ports: - "8080:8080" environment: - LISTMONK_API_URL=http://backend:9000 depends_on: - db volumes: - ../:/app networks: - listmonk-dev backend: build: context: ../ dockerfile: dev/app.Dockerfile command: ["make", "run-backend-docker"] ports: - "9000:9000" depends_on: - db volumes: - ../:/app - $GOPATH/pkg/mod/cache:/go/pkg/mod/cache networks: - listmonk-dev volumes: listmonk-dev-db: networks: listmonk-dev: ================================================ FILE: docker-compose.yml ================================================ # All LISTMONK_* env variables also support the LISTMONK_*_FILE pattern for loading secrets from files with Docker secrets and Podman # eg: LISTMONK_ADMIN_USER -> LISTMONK_ADMIN_USER_FILE=/path/to/file_with_value x-db-credentials: &db-credentials # Use the default POSTGRES_ credentials if they're available or simply default to "listmonk" POSTGRES_USER: &db-user listmonk # for database user, password, and database name POSTGRES_PASSWORD: &db-password listmonk POSTGRES_DB: &db-name listmonk services: # listmonk app app: image: listmonk/listmonk:latest container_name: listmonk_app restart: unless-stopped ports: - "9000:9000" # To change the externally exposed port, change to: $custom_port:9000 networks: - listmonk hostname: listmonk.example.com # Recommend using FQDN for hostname depends_on: - db command: [sh, -c, "./listmonk --install --idempotent --yes --config '' && ./listmonk --upgrade --yes --config '' && ./listmonk --config ''"] # --config (file) param is set to empty so that listmonk only uses the env vars (below) for config. # --install --idempotent ensures that DB installation happens only once on an empty DB, on the first ever start. # --upgrade automatically runs any DB migrations when a new image is pulled. environment: # The same params as in config.toml are passed as env vars here. LISTMONK_app__address: 0.0.0.0:9000 LISTMONK_db__user: *db-user LISTMONK_db__password: *db-password LISTMONK_db__database: *db-name LISTMONK_db__host: db LISTMONK_db__port: 5432 LISTMONK_db__ssl_mode: disable LISTMONK_db__max_open: 25 LISTMONK_db__max_idle: 25 LISTMONK_db__max_lifetime: 300s TZ: Etc/UTC LISTMONK_ADMIN_USER: ${LISTMONK_ADMIN_USER:-} # If these (optional) are set during the first `docker compose up`, then the Super Admin user is automatically created. LISTMONK_ADMIN_PASSWORD: ${LISTMONK_ADMIN_PASSWORD:-} # Otherwise, the user can be setup on the web app after the first visit to http://localhost:9000 volumes: - ./uploads:/listmonk/uploads:rw # Mount an uploads directory on the host to /listmonk/uploads inside the container. # To use this, change directory path in Admin -> Settings -> Media to /listmonk/uploads # Postgres database db: image: postgres:17-alpine container_name: listmonk_db restart: unless-stopped ports: - "127.0.0.1:5432:5432" # Only bind on the local interface. To connect to Postgres externally, change this to 0.0.0.0 networks: - listmonk environment: <<: *db-credentials healthcheck: test: ["CMD-SHELL", "pg_isready -U listmonk"] interval: 10s timeout: 5s retries: 6 volumes: - type: volume source: listmonk-data target: /var/lib/postgresql/data networks: listmonk: volumes: listmonk-data: ================================================ FILE: docker-entrypoint.sh ================================================ #!/bin/sh set -e export PUID=${PUID:-0} export PGID=${PGID:-0} export GROUP_NAME="app" export USER_NAME="app" # This function evaluates if the supplied PGID is already in use # if it is not in use, it creates the group with the PGID # if it is in use, it sets the GROUP_NAME to the existing group create_group() { if ! getent group ${PGID} > /dev/null 2>&1; then addgroup -g ${PGID} ${GROUP_NAME} else existing_group=$(getent group ${PGID} | cut -d: -f1) export GROUP_NAME=${existing_group} fi } # This function evaluates if the supplied PUID is already in use # if it is not in use, it creates the user with the PUID and PGID create_user() { if ! getent passwd ${PUID} > /dev/null 2>&1; then adduser -u ${PUID} -G ${GROUP_NAME} -s /bin/sh -D ${USER_NAME} else existing_user=$(getent passwd ${PUID} | cut -d: -f1) export USER_NAME=${existing_user} fi } # Run the needed functions to create the user and group create_group create_user load_secret_files() { # Save and restore IFS old_ifs="$IFS" IFS=' ' # Capture all env variables starting with LISTMONK_ and ending with _FILE. # It's value is assumed to be a file path with its actual value. for line in $(env | grep '^LISTMONK_.*_FILE='); do var="${line%%=*}" fpath="${line#*=}" # If it's a valid file, read its contents and assign it to the var # without the _FILE suffix. # Eg: LISTMONK_DB_USER_FILE=/run/secrets/user -> LISTMONK_DB_USER=$(contents of /run/secrets/user) if [ -f "$fpath" ]; then new_var="${var%_FILE}" export "$new_var"="$(cat "$fpath")" fi done IFS="$old_ifs" } # Load env variables from files if LISTMONK_*_FILE variables are set. load_secret_files # Try to set the ownership of the app directory to the app user. if ! chown -R ${PUID}:${PGID} /listmonk 2>/dev/null; then echo "Warning: Failed to change ownership of /listmonk. Readonly volume?" fi echo "Launching listmonk with user=[${USER_NAME}] group=[${GROUP_NAME}] PUID=[${PUID}] PGID=[${PGID}]" # If running as root and PUID is not 0, then execute command as PUID # this allows us to run the container as a non-root user if [ "$(id -u)" = "0" ] && [ "${PUID}" != "0" ]; then su-exec ${PUID}:${PGID} "$@" else exec "$@" fi ================================================ FILE: docs/README.md ================================================ # Static website and docs This repository contains the source for the static website https://listmonk.app - The website is in `site` and is built with hugo (run `hugo serve` inside `site` to preview). - Documentation is in `docs` and is built with mkdocs (inside `docs`, run `mkdocs serve` to preview after running `pip install -r requirements.txt`) - `i18n` directory has the static UI for i18n translations: https://listmonk.app/i18n ================================================ FILE: docs/docs/content/apis/apis.md ================================================ # APIs All features that are available on the listmonk dashboard are also available as REST-like HTTP APIs that can be interacted with directly. Request and response bodies are JSON. This allows easy scripting of listmonk and integration with other systems, for instance, synchronisation with external subscriber databases. !!! note If you come across API calls that are yet to be documented, please consider contributing to docs. ## Auth HTTP API requests support BasicAuth and a Authorization `token` headers. API users and tokens with the required permissions can be created and managed on the admin UI (Admin -> Users). ##### BasicAuth example ```shell curl -u "api_user:token" http://localhost:9000/api/lists ``` ##### Authorization token example ```shell curl -H "Authorization: token api_user:token" http://localhost:9000/api/lists ``` ## Permissions **User role**: Permissions allowed for a user are defined as a *User role* (Admin -> User roles) and then attached to a user. **List role**: Read / write permissions per-list can be defined as a *List role* (Admin -> User roles) and then attached to a user. In a *User role*, `lists:get_all` or `lists:manage_all` permission supercede and override any list specific permissions for a user defined in a *List role*. To manage lists and subscriber list subscriptions via API requests, ensure that the appropriate permissions are attached to the API user. ______________________________________________________________________ ## Response structure ### Successful request ```http HTTP/1.1 200 OK Content-Type: application/json { "data": {} } ``` All responses from the API server are JSON with the content-type application/json unless explicitly stated otherwise. A successful 200 OK response always has a JSON response body with a status key with the value success. The data key contains the full response payload. ### Failed request ```http HTTP/1.1 500 Server error Content-Type: application/json { "message": "Error message" } ``` A failure response is preceded by the corresponding 40x or 50x HTTP header. There may be an optional `data` key with additional payload. ### Timestamps All timestamp fields are in the format `2019-01-01T09:00:00.000000+05:30`. The seconds component is suffixed by the milliseconds, followed by the `+` and the timezone offset. ### Common HTTP error codes | Code | | | ----- | ----------------------------------------------------------------------------| | 400 | Missing or bad request parameters or values | | 403 | Session expired or invalidate. Must relogin | | 404 | Request resource was not found | | 405 | Request method (GET, POST etc.) is not allowed on the requested endpoint | | 410 | The requested resource is gone permanently | | 422 | Unprocessable entity. Unable to process request as it contains invalid data | | 429 | Too many requests to the API (rate limiting) | | 500 | Something unexpected went wrong | | 502 | The backend OMS is down and the API is unable to communicate with it | | 503 | Service unavailable; the API is down | | 504 | Gateway timeout; the API is unreachable | ## OpenAPI (Swagger) spec The auto-generated OpenAPI (Swagger) specification site for the APIs are available at [**listmonk.app/docs/swagger**](https://listmonk.app/docs/swagger/) ================================================ FILE: docs/docs/content/apis/bounces.md ================================================ # API / Bounces Method | Endpoint | Description ---------|---------------------------------------------------------|------------------------------------------------ GET | [/api/bounces](#get-apibounces) | Retrieve bounce records. DELETE | [/api/bounces](#delete-apibounces) | Delete all/multiple bounce records. DELETE | [/api/bounces/{bounce_id}](#delete-apibouncesbounce_id) | Delete specific bounce record. ______________________________________________________________________ #### GET /api/bounces Retrieve the bounce records. ##### Parameters | Name | Type | Required | Description | |:-----------|:---------|:---------|:-----------------------------------------------------------------| | campaign_id| number | | Bounce record retrieval for particular campaign id | | page | number | | Page number for pagination. | | per_page | number | | Results per page. Set to 'all' to return all results. | | source | string | | | | order_by | string | | Fields by which bounce records are ordered. Options:"email", "campaign_name", "source", "created_at". | | order | number | | Sorts the result. Allowed values: 'asc','desc' | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/bounces?campaign_id=1&page=1&per_page=2' \ -H 'accept: application/json' -H 'Content-Type: application/x-www-form-urlencoded' \ --data '{"source":"demo","order_by":"created_at","order":"asc"}' ``` ##### Example Response ```json { "data": { "results": [ { "id": 839971, "type": "hard", "source": "demo", "meta": { "some": "parameter" }, "created_at": "2024-08-20T23:54:22.851858Z", "email": "gilles.deleuze@example.app", "subscriber_uuid": "32ca1f3e-1a1d-42e1-af04-df0757f420f3", "subscriber_id": 60, "campaign": { "id": 1, "name": "Test campaign" } }, { "id": 839725, "type": "hard", "source": "demo", "meta": { "some": "parameter" }, "created_at": "2024-08-20T22:46:36.393547Z", "email": "gottfried.leibniz@example.app", "subscriber_uuid": "5911d3f4-2346-4bfc-aad2-eb319ab0e879", "subscriber_id": 13, "campaign": { "id": 1, "name": "Test campaign" } } ], "query": "", "total": 528, "per_page": 2, "page": 1 } } ``` ______________________________________________________________________ #### DELETE /api/bounces To delete all bounces. ##### Parameters | Name | Type | Required | Description | |:--------|:----------|:---------|:-------------------------------------| | all | bool | Yes | Bool to confirm deleting all bounces | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/bounces?all=true' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/bounces To delete multiple bounce records. ##### Parameters | Name | Type | Required | Description | |:--------|:----------|:---------|:-------------------------------------| | id | number | Yes | Id's of bounce records to delete. | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/bounces?id=840965&id=840168&id=840879' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/bounces/{bounce_id} To delete specific bounce id. ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/bounces/840965' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/campaigns.md ================================================ # API / Campaigns | Method | Endpoint | Description | | :----- | :-------------------------------------------------------------------------- | :---------------------------------------- | | GET | [/api/campaigns](#get-apicampaigns) | Retrieve all campaigns. | | GET | [/api/campaigns/{campaign_id}](#get-apicampaignscampaign_id) | Retrieve a specific campaign. | | GET | [/api/campaigns/{campaign_id}/preview](#get-apicampaignscampaign_idpreview) | Retrieve preview of a campaign. | | GET | [/api/campaigns/running/stats](#get-apicampaignsrunningstats) | Retrieve stats of specified campaigns. | | GET | [/api/campaigns/analytics/{type}](#get-apicampaignsanalyticstype) | Retrieve view counts for a campaign. | | POST | [/api/campaigns](#post-apicampaigns) | Create a new campaign. | | POST | [/api/campaigns/{campaign_id}/test](#post-apicampaignscampaign_idtest) | Test campaign with arbitrary subscribers. | | PUT | [/api/campaigns/{campaign_id}](#put-apicampaignscampaign_id) | Update a campaign. | | PUT | [/api/campaigns/{campaign_id}/status](#put-apicampaignscampaign_idstatus) | Change status of a campaign. | | PUT | [/api/campaigns/{campaign_id}/archive](#put-apicampaignscampaign_idarchive) | Publish campaign to public archive. | | DELETE | [/api/campaigns/{campaign_id}](#delete-apicampaignscampaign_id) | Delete a campaign. | | DELETE | [/api/campaigns](#delete-apicampaigns) | Delete multiple campaigns. | ____________________________________________________________________________________________________________________________________ #### GET /api/campaigns Retrieve all campaigns. ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns?page=1&per_page=100' ``` ##### Parameters | Name | Type | Required | Description | | :------- | :------- | :------- | :----------------------------------------------------------------------- | | order | string | | Sorting order: ASC for ascending, DESC for descending. | | order_by | string | | Result sorting field. Options: name, status, created_at, updated_at. | | query | string | | String to filtter by campaign name and subject (fulltext and substring). | | status | []string | | Status to filter campaigns. Repeat in the query for multiple values. | | tags | []string | | Tags to filter campaigns. Repeat in the query for multiple values. | | page | number | | Page number for paginated results. | | per_page | number | | Results per page. Set as 'all' for all results. | | no_body | boolean | | When set to true, returns response without body content. | ##### Example Response ```json { "data": { "results": [ { "id": 1, "created_at": "2020-03-14T17:36:41.29451+01:00", "updated_at": "2020-03-14T17:36:41.29451+01:00", "views": 0, "clicks": 0, "lists": [ { "id": 1, "name": "Default list" } ], "started_at": null, "to_send": 0, "sent": 0, "uuid": "57702beb-6fae-4355-a324-c2fd5b59a549", "type": "regular", "name": "Test campaign", "subject": "Welcome to listmonk", "from_email": "No ReplyHi {{ .Subscriber.FirstName }}!
\n\t\t\tThis is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.", "body_source": null, "send_at": "2020-03-15T17:36:41.293233+01:00", "status": "draft", "content_type": "richtext", "tags": [ "test-campaign" ], "template_id": 1, "messenger": "email" } ], "query": "", "total": 1, "per_page": 20, "page": 1 } } ``` ______________________________________________________________________ #### GET /api/campaigns/{campaign_id} Retrieve a specific campaign. ##### Parameters | Name | Type | Required | Description | | :---------- | :------ | :------- | :------------------------------------------------------- | | campaign_id | number | Yes | Campaign ID. | | no_body | boolean | | When set to true, returns response without body content. | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns/1' ``` ##### Example Response ```json { "data": { "id": 1, "created_at": "2020-03-14T17:36:41.29451+01:00", "updated_at": "2020-03-14T17:36:41.29451+01:00", "views": 0, "clicks": 0, "lists": [ { "id": 1, "name": "Default list" } ], "started_at": null, "to_send": 0, "sent": 0, "uuid": "57702beb-6fae-4355-a324-c2fd5b59a549", "type": "regular", "name": "Test campaign", "subject": "Welcome to listmonk", "from_email": "No ReplyHi {{ .Subscriber.FirstName }}!
\n\t\t\tThis is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.", "body_source": null, "send_at": "2020-03-15T17:36:41.293233+01:00", "status": "draft", "content_type": "richtext", "tags": [ "test-campaign" ], "template_id": 1, "messenger": "email" } } ``` ______________________________________________________________________ #### GET /api/campaigns/{campaign_id}/preview Preview a specific campaign. ##### Parameters | Name | Type | Required | Description | | :---------- | :----- | :------- | :---------------------- | | campaign_id | number | Yes | Campaign ID to preview. | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns/1/preview' ``` ##### Example Response ```htmlHi John!
This is a test e-mail campaign. Your second name is Doe and you are from Bengaluru. ``` ______________________________________________________________________ #### GET /api/campaigns/running/stats Retrieve stats of specified campaigns. ##### Parameters | Name | Type | Required | Description | | :---------- | :----- | :------- | :----------------------------- | | campaign_id | number | Yes | Campaign IDs to get stats for. | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns/running/stats?campaign_id=1' ``` ##### Example Response ```json { "data": [] } ``` ______________________________________________________________________ #### GET /api/campaigns/analytics/{type} Retrieve stats of specified campaigns. ##### Parameters | Name | Type | Required | Description | | :--- | :--------- | :------- | :-------------------------------------------- | | id | number\[\] | Yes | Campaign IDs to get stats for. | | type | string | Yes | Analytics type: views, links, clicks, bounces | | from | string | Yes | Start value of date range. | | to | string | Yes | End value of date range. | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns/analytics/views?id=1&from=2024-08-04&to=2024-08-12' ``` ##### Example Response ```json { "data": [ { "campaign_id": 1, "count": 10, "timestamp": "2024-08-04T00:00:00Z" }, { "campaign_id": 1, "count": 14, "timestamp": "2024-08-08T00:00:00Z" }, { "campaign_id": 1, "count": 20, "timestamp": "2024-08-09T00:00:00Z" }, { "campaign_id": 1, "count": 21, "timestamp": "2024-08-10T00:00:00Z" }, { "campaign_id": 1, "count": 21, "timestamp": "2024-08-11T00:00:00Z" } ] } ``` ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/campaigns/analytics/links?id=1&from=2024-08-04T18%3A30%3A00.624Z&to=2024-08-12T18%3A29%3A00.624Z' ``` ##### Example Response ```json { "data": [ { "url": "https://freethebears.org", "count": 294 }, { "url": "https://calmcode.io", "count": 278 }, { "url": "https://climate.nasa.gov", "count": 261 }, { "url": "https://www.storybreathing.com", "count": 260 } ] } ``` ______________________________________________________________________ #### POST /api/campaigns Create a new campaign. ##### Parameters | Name | Type | Required | Description | | :----------- | :--------- | :------- | :--------------------------------------------------------------------------------------------------------------------- | | name | string | Yes | Campaign name. | | subject | string | Yes | Campaign email subject. | | lists | number\[\] | Yes | List IDs to send campaign to. | | from_email | string | | 'From' email in campaign emails. Defaults to value from settings if not provided. | | type | string | Yes | Campaign type: 'regular' or 'optin'. | | content_type | string | Yes | Content type: 'richtext', 'html', 'markdown', 'plain', 'visual'. | | body | string | Yes | Content body of campaign. | | body_source | string | | If content_type is `visual`, the JSON block source of the body. | | altbody | string | | Alternate plain text body for HTML (and richtext) emails. | | send_at | string | | Timestamp to schedule campaign. Format: 'YYYY-MM-DDTHH:MM:SSZ'. | | messenger | string | | 'email' or a custom messenger defined in settings. Defaults to 'email' if not provided. | | template_id | number | | Template ID to use. Defaults to default template if not provided. | | tags | string\[\] | | Tags to mark campaign. | | headers | JSON | | Key-value pairs to send as SMTP headers. Example: \[{"x-custom-header": "value"}\]. | | attribs | JSON | | Optional JSON object attributes that can be used in the campaign message template. Example `{"location": "Somewhere"}` | ##### Example request ```shell curl -u "api_user:token" 'http://localhost:9000/api/campaigns' -X POST -H 'Content-Type: application/json;charset=utf-8' --data-raw '{"name":"Test campaign","subject":"Hello, world","lists":[1],"from_email":"listmonkHi {{ .Subscriber.FirstName }}!
\n\t\t\tThis is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.", "send_at": "2020-03-15T17:36:41.293233+01:00", "status": "scheduled", "content_type": "richtext", "tags": [ "test-campaign" ], "template_id": 1, "messenger": "email" } } ``` ______________________________________________________________________ #### PUT /api/campaigns/{campaign_id}/archive Publish campaign to public archive. ##### Parameters | Name | Type | Required | Description | | :------------------ | :---------- | :------- | :------------------------------------------------------------------------ | | campaign_id | number | Yes | Campaign ID to publish to public archive. | | archive | bool | Yes | State of the public archive. | | archive_template_id | number | No | Archive template id. Defaults to 0. | | archive_meta | JSON string | No | Optional Metadata to use in campaign message or template.Eg: name, email. | | archive_slug | string | No | Name for page to be used in public archive URL | ##### Example Request ```shell curl -u "api_user:token" -X PUT 'http://localhost:8080/api/campaigns/33/archive' --header 'Content-Type: application/json' --data-raw '{"archive":true,"archive_template_id":1,"archive_meta":{},"archive_slug":"my-newsletter-old-edition"}' ``` ##### Example Response ```json { "data": { "archive": true, "archive_template_id": 1, "archive_meta": {}, "archive_slug": "my-newsletter-old-edition" } } ``` ______________________________________________________________________ #### DELETE /api/campaigns/{campaign_id} Delete a campaign. ##### Parameters | Name | Type | Required | Description | | :---------- | :----- | :------- | :--------------------- | | campaign_id | number | Yes | Campaign ID to delete. | ##### Example Request ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/campaigns/34' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/campaigns Delete multiple campaigns by IDs or by a search query. ##### Parameters | Name | Type | Required | Description | | :---- | :--------- | :---------------------------- | :-------------------------------------------------------------------------- | | id | number\[\] | Yes (if `query` not provided) | Onr or more campaign IDs to delete. | | query | string | Yes (if `id` not provided) | Fulltext search query to filter campaigns for deletion (same as GET query). | ##### Example Request (by IDs) ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/campaigns?id=10&id=11&id=12' ``` ##### Example Request (by search query) ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/campaigns?query=test%20campaign' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/import.md ================================================ # API / Import Method | Endpoint | Description ---------|-------------------------------------------------|------------------------------------------------ GET | [/api/import/subscribers](#get-apiimportsubscribers) | Retrieve import statistics. GET | [/api/import/subscribers/logs](#get-apiimportsubscriberslogs) | Retrieve import logs. POST | [/api/import/subscribers](#post-apiimportsubscribers) | Upload a file for bulk subscriber import. DELETE | [/api/import/subscribers](#delete-apiimportsubscribers) | Stop and remove an import. ______________________________________________________________________ #### GET /api/import/subscribers Retrieve the status of an ongoing import. ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/import/subscribers' ``` ##### Example Response ```json { "data": { "name": "", "total": 0, "imported": 0, "status": "none" } } ``` ______________________________________________________________________ #### GET /api/import/subscribers/logs Retrieve logs from an ongoing import. ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/import/subscribers/logs' ``` ##### Example Response ```json { "data": "2020/04/08 21:55:20 processing 'import.csv'\n2020/04/08 21:55:21 imported finished\n" } ``` ______________________________________________________________________ #### POST /api/import/subscribers Send a CSV (optionally ZIP compressed) file to import subscribers. Use a multipart form POST. ##### Parameters | Name | Type | Required | Description | |:-------|:------------|:---------|:-----------------------------------------| | params | JSON string | Yes | Stringified JSON with import parameters. | | file | file | Yes | File for upload. | #### `params` (JSON string) | Name | Type | Required | Description | |:----------|:---------|:---------|:-----------------------------------------------------------------------------------------------------------------------------------| | mode | string | Yes | `subscribe` or `blocklist` | | delim | string | Yes | Single character indicating delimiter used in the CSV file, eg: `,` | | lists | []number | | Array of list IDs to subscribe to. | | overwrite | bool | | Whether to overwrite the subscriber parameters including subscriptions or ignore records that are already present in the database. | ##### Example Request ```shell curl -u "api_user:token" -X POST 'http://localhost:9000/api/import/subscribers' \ -F 'params={"mode":"subscribe", "subscription_status":"confirmed", "delim":",", "lists":[1, 2], "overwrite": true}' \ -F "file=@/path/to/subs.csv" ``` ##### Example Response ```json { "mode": "subscribe", // subscribe or blocklist "delim": ",", // delimiter in the uploaded file "lists":[1], // array of list IDs to import into "overwrite": true // overwrite existing entries or skip them? } ``` ______________________________________________________________________ #### DELETE /api/import/subscribers Stop and delete an ongoing import. ##### Example Request ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/import/subscribers' ``` ##### Example Response ```json { "data": { "name": "", "total": 0, "imported": 0, "status": "none" } } ``` ================================================ FILE: docs/docs/content/apis/lists.md ================================================ # API / Lists | Method | Endpoint | Description | | :----- | :---------------------------------------------- | :------------------------ | | GET | [/api/lists](#get-apilists) | Retrieve all lists. | | GET | [/api/public/lists](#get-public-apilists) | Retrieve public lists. | | GET | [/api/lists/{list_id}](#get-apilistslist_id) | Retrieve a specific list. | | POST | [/api/lists](#post-apilists) | Create a new list. | | PUT | [/api/lists/{list_id}](#put-apilistslist_id) | Update a list. | | DELETE | [/api/lists/{list_id}](#delete-apilistslist_id) | Delete a list. | | DELETE | [/api/lists](#delete-apilists) | Delete multiple lists. | ______________________________________________________________________ #### GET /api/lists Retrieve lists. > **Note:** Lists with `status: archived` are hidden from list selectors in campaigns, public subscription forms, and roles by default. They can only be viewed by filtering with `status=archived` or by viewing all lists without a status filter. ##### Parameters | Name | Type | Required | Description | | :------- | :------- | :------- | :------------------------------------------------------------------------------------------------- | | query | string | | String for list name search. | | status | string | | Status to filter lists. Options: active, archived. Defaults to showing all lists if not specified. | | minimal | boolean | | If true, returns lists without subscriber counts (faster). Defaults to false. | | tag | []string | | Tags to filter lists. Repeat in the query for multiple values. | | order_by | string | | Sort field. Options: name, status, created_at, updated_at. | | order | string | | Sorting order. Options: ASC, DESC. | | page | number | | Page number for pagination. | | per_page | number | | Results per page. Set to 'all' to return all results. | ##### Example Request ```shell # Get all lists curl -u "api_user:token" -X GET 'http://localhost:9000/api/lists?page=1&per_page=100' # Get only active lists curl -u "api_user:token" -X GET 'http://localhost:9000/api/lists?status=active&per_page=100' # Get archived lists with minimal data curl -u "api_user:token" -X GET 'http://localhost:9000/api/lists?status=archived&minimal=true&per_page=all' ``` ##### Example Response ```json { "data": { "results": [ { "id": 1, "created_at": "2020-02-10T23:07:16.194843+01:00", "updated_at": "2020-03-06T22:32:01.118327+01:00", "uuid": "ce13e971-c2ed-4069-bd0c-240e9a9f56f9", "name": "Default list", "type": "public", "optin": "double", "status": "active", "tags": [ "test" ], "subscriber_count": 2 }, { "id": 2, "created_at": "2020-03-04T21:12:09.555013+01:00", "updated_at": "2020-03-06T22:34:46.405031+01:00", "uuid": "f20a2308-dfb5-4420-a56d-ecf0618a102d", "name": "get", "type": "private", "optin": "single", "status": "active", "tags": [], "subscriber_count": 0 } ], "total": 5, "per_page": 20, "page": 1 } } ``` ______________________________________________________________________ #### GET /api/public/lists Retrieve public lists with name and uuid to submit a subscription. This is an unauthenticated call to enable scripting to subscription form. > **Note:** This endpoint only returns lists with `type: public` and `status: active`. Archived lists are never shown on public subscription forms. ##### Example Request ```shell curl -X GET 'http://localhost:9000/api/public/lists' ``` ##### Example Response ```json [ { "uuid": "55e243af-80c6-4169-8d7f-bc571e0269e9", "name": "Opt-in list" } ] ``` ______________________________________________________________________ #### GET /api/lists/{list_id} Retrieve a specific list. ##### Parameters | Name | Type | Required | Description | | :------ | :----- | :------- | :-------------------------- | | list_id | number | Yes | ID of the list to retrieve. | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/lists/5' ``` ##### Example Response ```json { "data": { "id": 5, "created_at": "2020-03-07T06:31:06.072483+01:00", "updated_at": "2020-03-07T06:31:06.072483+01:00", "uuid": "1bb246ab-7417-4cef-bddc-8fc8fc941d3a", "name": "Test list", "type": "public", "optin": "double", "status": "active", "tags": [], "subscriber_count": 0 } } ``` ______________________________________________________________________ #### POST /api/lists Create a new list. ##### Parameters | Name | Type | Required | Description | | :---------- | :--------- | :------- | :----------------------------------------------------------------- | | name | string | Yes | Name of the new list. | | type | string | Yes | Type of list. Options: private, public. | | optin | string | Yes | Opt-in type. Options: single, double. | | status | string | No | Status of the list. Options: active, archived. Defaults to active. | | tags | string\[\] | | Associated tags for a list. | | description | string | No | Description of the new list. | ##### Example Request ```shell curl -u "api_user:token" -X POST 'http://localhost:9000/api/lists' ``` ##### Example Response ```json { "data": { "id": 5, "created_at": "2020-03-07T06:31:06.072483+01:00", "updated_at": "2020-03-07T06:31:06.072483+01:00", "uuid": "1bb246ab-7417-4cef-bddc-8fc8fc941d3a", "name": "Test list", "type": "public", "optin": "single", "status": "active", "tags": [], "subscriber_count": 0, "description": "This is a test list" } } ``` ______________________________________________________________________ #### PUT /api/lists/{list_id} Update a list. ##### Parameters | Name | Type | Required | Description | | :---------- | :--------- | :------- | :--------------------------------------------- | | list_id | number | Yes | ID of the list to update. | | name | string | | New name for the list. | | type | string | | Type of list. Options: private, public. | | optin | string | | Opt-in type. Options: single, double. | | status | string | | Status of the list. Options: active, archived. | | tags | string\[\] | | Associated tags for the list. | | description | string | | Description of the list. | ##### Example Request ```shell curl -u "api_user:token" -X PUT 'http://localhost:9000/api/lists/5' \ --form 'name=modified test list' \ --form 'type=private' ``` ##### Example Response ```json { "data": { "id": 5, "created_at": "2020-03-07T06:31:06.072483+01:00", "updated_at": "2020-03-07T06:52:15.208075+01:00", "uuid": "1bb246ab-7417-4cef-bddc-8fc8fc941d3a", "name": "modified test list", "type": "private", "optin": "single", "status": "active", "tags": [], "subscriber_count": 0, "description": "This is a test list" } } ``` ______________________________________________________________________ #### DELETE /api/lists/{list_id} Delete a specific list. ##### Parameters | Name | Type | Required | Description | | :------ | :----- | :------- | :------------------------ | | list_id | Number | Yes | ID of the list to delete. | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/lists/1' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/lists Delete multiple lists by IDs or by a search query. > **Note:** Users can only delete lists they have `manage` permission for. Any lists in the query that the user doesn't have permission to manage is ignored. ##### Parameters | Name | Type | Required | Description | | :---- | :--------- | :---------------------------- | :----------------------------------------------------------------- | | id | number\[\] | Yes (if `query` not provided) | One or more list IDs to delete. | | query | string | Yes (if `id` not provided) | Search query to filter lists for deletion (same as the GET query). | ##### Example Request (by IDs) ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/lists?id=10&id=11&id=12' ``` ##### Example Request (by search query) ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/lists?query=test%20list' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/media.md ================================================ # API / Media Method | Endpoint | Description -------|------------------------------------------------------|--------------------------------- GET | [/api/media](#get-apimedia) | Get uploaded media file GET | [/api/media/{media_id}](#get-apimediamedia_id) | Get specific uploaded media file POST | [/api/media](#post-apimedia) | Upload media file DELETE | [/api/media/{media_id}](#delete-apimediamedia_id) | Delete uploaded media file ______________________________________________________________________ #### GET /api/media Get an uploaded media file. ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/media' \ --header 'Content-Type: multipart/form-data; boundary=--------------------------093715978792575906250298' ``` ##### Example Response ```json { "data": [ { "id": 1, "uuid": "ec7b45ce-1408-4e5c-924e-965326a20287", "filename": "Media file", "created_at": "2020-04-08T22:43:45.080058+01:00", "thumb_url": "/uploads/image_thumb.jpg", "uri": "/uploads/image.jpg" } ] } ``` ______________________________________________________________________ #### GET /api/media/{media_id} Retrieve a specific media. ##### Parameters | Name | Type | Required | Description | |:--------------|:----------|:---------|:-----------------| | media_id | Number | Yes | Media ID. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/media/7' ``` ##### Example Response ```json { "data": { "id": 7, "uuid": "62e32e97-d6ca-4441-923f-b62607000dd1", "filename": "ResumeB.pdf", "content_type": "application/pdf", "created_at": "2024-08-06T11:28:53.888257+05:30", "thumb_url": null, "provider": "filesystem", "meta": {}, "url": "http://localhost:9000/uploads/ResumeB.pdf" } } ``` ______________________________________________________________________ #### POST /api/media Upload a media file. ##### Parameters | Field | Type | Required | Description | |-------|-----------|----------|---------------------| | file | File | Yes | Media file to upload| ##### Example Request ```shell curl -u "api_user:token" -X POST 'http://localhost:9000/api/media' \ --header 'Content-Type: multipart/form-data; boundary=--------------------------183679989870526937212428' \ --form 'file=@/path/to/image.jpg' ``` ##### Example Response ```json { "data": { "id": 1, "uuid": "ec7b45ce-1408-4e5c-924e-965326a20287", "filename": "Media file", "created_at": "2020-04-08T22:43:45.080058+01:00", "thumb_uri": "/uploads/image_thumb.jpg", "uri": "/uploads/image.jpg" } } ``` ______________________________________________________________________ #### DELETE /api/media/{media_id} Delete an uploaded media file. ##### Parameters | Field | Type | Required | Description | |----------|-----------|----------|-------------------------| | media_id | number | Yes | ID of media file to delete | ##### Example Request ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/media/1' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/sdks.md ================================================ # SDKs and client libraries A list of 3rd party client libraries and SDKs that have been written for listmonk APIs. !!! note The list is community sourced. They have not been verified and are not officially supported. - [WordPress - WooCommerce plugin](https://github.com/post-duif/integration-listmonk-wordpress-plugin) integration for listmonk - [listmonk ](https://github.com/mikeckennedy/listmonk) — Python API client - [listmonk-api](https://github.com/Knuckles-Team/listmonk-api) — Python API client - [frappe_listmonk](https://github.com/anandology/frappe_listmonk) — Frappe framework integration for listmonk - [auto-newsletter-listmonk](https://github.com/chaddyc/auto-newsletter-listmonk) — Ghost CMS integration - [listmonk-newsletter](https://github.com/iloveitaly/listmonk-newsletter) - RSS to listmonk integration for email newsletters - [listmonk-crysctal](https://github.com/russ/listmonk-crystal) — Crystal lang API client - [terraform-provider-listmonk](https://github.com/Muravlev/terraform-provider-listmonk) — Manage listmonk templates in Terraform - [listmonk-php-client](https://github.com/arunnabraham/listmonk-php-client) — PHP API client - [php-listmonk](https://github.com/junisan/php-listmonk) — PHP API client - [go-listmonk](https://github.com/EzeXchange-API/go-listmonk) — Go API client - [listmonk-nodejs-api](https://github.com/mihairaulea/listmonk-nodejs-api) — NodeJS API client - [@maloma/listmonk](https://codeberg.org/maloma/listmonk-sdk) — NodeJS/Bun Typescript SDK - [listmonk-laravel](https://github.com/theafolayan/listmonk-laravel) — Laravel API Client - [nuxt-listmonk](https://github.com/roncallyt/nuxt-listmonk) — Listmonk module for Nuxt.js - [listmonk-japi](https://codeberg.org/hlassiege/listmonk-japi) - Listmonk client for Java/kotlin - [listmonk-mcp](https://github.com/rhnvrm/listmonk-mcp) — MCP (Model Context Protocol) server for Claude integration - [N8N Nodes](https://github.com/wiesinghilker/n8n-nodes-listmonk) — Adds Listmonk Nodes for N8N Automations ================================================ FILE: docs/docs/content/apis/subscribers.md ================================================ # API / Subscribers | Method | Endpoint | Description | | ------ | --------------------------------------------------------------------------------------- | ---------------------------------------------- | | GET | [/api/subscribers](#get-apisubscribers) | Query and retrieve subscribers. | | GET | [/api/subscribers/{subscriber_id}](#get-apisubscriberssubscriber_id) | Retrieve a specific subscriber. | | GET | [/api/subscribers/{subscriber_id}/export](#get-apisubscriberssubscriber_idexport) | Export a specific subscriber. | | GET | [/api/subscribers/{subscriber_id}/bounces](#get-apisubscriberssubscriber_idbounces) | Retrieve a subscriber bounce records. | | POST | [/api/subscribers](#post-apisubscribers) | Create a new subscriber. | | POST | [/api/subscribers/{subscriber_id}/optin](#post-apisubscriberssubscriber_idoptin) | Sends optin confirmation email to subscribers. | | POST | [/api/public/subscription](#post-apipublicsubscription) | Create a public subscription. | | PUT | [/api/subscribers/lists](#put-apisubscriberslists) | Modify subscriber list memberships. | | PUT | [/api/subscribers/{subscriber_id}](#put-apisubscriberssubscriber_id) | Update a specific subscriber. | | PUT | [/api/subscribers/{subscriber_id}/blocklist](#put-apisubscriberssubscriber_idblocklist) | Blocklist a specific subscriber. | | PUT | [/api/subscribers/blocklist](#put-apisubscribersblocklist) | Blocklist one or many subscribers. | | PUT | [/api/subscribers/query/blocklist](#put-apisubscribersqueryblocklist) | Blocklist subscribers based on SQL expression. | | DELETE | [/api/subscribers/{subscriber_id}](#delete-apisubscriberssubscriber_id) | Delete a specific subscriber. | | DELETE | [/api/subscribers/{subscriber_id}/bounces](#delete-apisubscriberssubscriber_idbounces) | Delete a specific subscriber's bounce records. | | DELETE | [/api/subscribers](#delete-apisubscribers) | Delete one or more subscribers. | | POST | [/api/subscribers/query/delete](#post-apisubscribersquerydelete) | Delete subscribers based on SQL expression. | ______________________________________________________________________ #### GET /api/subscribers Retrieve all subscribers. ##### Query parameters | Name | Type | Required | Description | | :------------------ | :----- | :------- | :-------------------------------------------------------------------- | | query | string | | Subscriber search by SQL expression. | | list_id | int[] | | ID of lists to filter by. Repeat in the query for multiple values. | | subscription_status | string | | Subscription status to filter by if there are one or more `list_id`s. | | order_by | string | | Result sorting field. Options: name, status, created_at, updated_at. | | order | string | | Sorting order: ASC for ascending, DESC for descending. | | page | number | | Page number for paginated results. | | per_page | number | | Results per page. Set as 'all' for all results. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers?page=1&per_page=100' ``` ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers?list_id=1&list_id=2&page=1&per_page=100' ``` ```shell curl -u 'api_username:access_token' -X GET 'http://localhost:9000/api/subscribers' \ --url-query 'page=1' \ --url-query 'per_page=100' \ --url-query "query=subscribers.name LIKE 'Test%' AND subscribers.attribs->>'city' = 'Bengaluru'" ``` ##### Example Response ```json { "data": { "results": [ { "id": 1, "created_at": "2020-02-10T23:07:16.199433+01:00", "updated_at": "2020-02-10T23:07:16.199433+01:00", "uuid": "ea06b2e7-4b08-4697-bcfc-2a5c6dde8f1c", "email": "john@example.com", "name": "John Doe", "attribs": { "city": "Bengaluru", "good": true, "type": "known" }, "status": "enabled", "lists": [ { "subscription_status": "unconfirmed", "id": 1, "uuid": "ce13e971-c2ed-4069-bd0c-240e9a9f56f9", "name": "Default list", "type": "public", "tags": [ "test" ], "created_at": "2020-02-10T23:07:16.194843+01:00", "updated_at": "2020-02-10T23:07:16.194843+01:00" } ] }, { "id": 2, "created_at": "2020-02-18T21:10:17.218979+01:00", "updated_at": "2020-02-18T21:10:17.218979+01:00", "uuid": "ccf66172-f87f-4509-b7af-e8716f739860", "email": "quadri@example.com", "name": "quadri", "attribs": {}, "status": "enabled", "lists": [ { "subscription_status": "unconfirmed", "id": 1, "uuid": "ce13e971-c2ed-4069-bd0c-240e9a9f56f9", "name": "Default list", "type": "public", "tags": [ "test" ], "created_at": "2020-02-10T23:07:16.194843+01:00", "updated_at": "2020-02-10T23:07:16.194843+01:00" } ] }, { "id": 3, "created_at": "2020-02-19T19:10:49.36636+01:00", "updated_at": "2020-02-19T19:10:49.36636+01:00", "uuid": "5d940585-3cc8-4add-b9c5-76efba3c6edd", "email": "sugar@example.com", "name": "sugar", "attribs": {}, "status": "enabled", "lists": [] } ], "query": "", "total": 3, "per_page": 20, "page": 1 } } ``` ______________________________________________________________________ #### GET /api/subscribers/{subscriber_id} Retrieve a specific subscriber. ##### Parameters | Name | Type | Required | Description | | :------------ | :----- | :------- | :--------------- | | subscriber_id | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers/1' ``` ##### Example Response ```json { "data": { "id": 1, "created_at": "2020-02-10T23:07:16.199433+01:00", "updated_at": "2020-02-10T23:07:16.199433+01:00", "uuid": "ea06b2e7-4b08-4697-bcfc-2a5c6dde8f1c", "email": "john@example.com", "name": "John Doe", "attribs": { "city": "Bengaluru", "good": true, "type": "known" }, "status": "enabled", "lists": [ { "subscription_status": "unconfirmed", "id": 1, "uuid": "ce13e971-c2ed-4069-bd0c-240e9a9f56f9", "name": "Default list", "type": "public", "tags": [ "test" ], "created_at": "2020-02-10T23:07:16.194843+01:00", "updated_at": "2020-02-10T23:07:16.194843+01:00" } ] } } ``` ______________________________________________________________________ #### GET /api/subscribers/{subscriber_id}/export Export a specific subscriber data that gives profile, list subscriptions, campaign views and link clicks information. Names of private lists are replaced with "Private list". ##### Parameters | Name | Type | Required | Description | | :------------ | :----- | :------- | :--------------- | | subscriber_id | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers/1/export' ``` ##### Example Response ```json { "profile": [ { "id": 1, "uuid": "c2cc0b31-b485-4d72-8ce8-b47081beadec", "email": "john@example.com", "name": "John Doe", "attribs": { "city": "Bengaluru", "good": true, "type": "known" }, "status": "enabled", "created_at": "2024-07-29T11:01:31.478677+05:30", "updated_at": "2024-07-29T11:01:31.478677+05:30" } ], "subscriptions": [ { "subscription_status": "unconfirmed", "name": "Private list", "type": "private", "created_at": "2024-07-29T11:01:31.478677+05:30" } ], "campaign_views": [], "link_clicks": [] } ``` ______________________________________________________________________ #### GET /api/subscribers/{subscriber_id}/bounces Get a specific subscriber bounce records. ##### Parameters | Name | Type | Required | Description | | :------------ | :----- | :------- | :--------------- | | subscriber_id | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers/1/bounces' ``` ##### Example Response ```json { "data": [ { "id": 841706, "type": "hard", "source": "demo", "meta": { "some": "parameter" }, "created_at": "2024-08-22T09:05:12.862877Z", "email": "thomas.hobbes@example.com", "subscriber_uuid": "137c0d83-8de6-44e2-a55f-d4238ab21969", "subscriber_id": 99, "campaign": { "id": 2, "name": "Welcome to listmonk" } }, { "id": 841680, "type": "hard", "source": "demo", "meta": { "some": "parameter" }, "created_at": "2024-08-19T14:07:53.141917Z", "email": "thomas.hobbes@example.com", "subscriber_uuid": "137c0d83-8de6-44e2-a55f-d4238ab21969", "subscriber_id": 99, "campaign": { "id": 1, "name": "Test campaign" } } ] } ``` ______________________________________________________________________ #### POST /api/subscribers Create a new subscriber. ##### Parameters | Name | Type | Required | Description | |:-------------------------|:-----------|:---------|:------------------------------------------------------------------------------------------------------------------------------| | email | string | Yes | Subscriber's email address. | | name | string | Yes | Subscriber's name. | | status | string | Yes | Subscriber's status: `enabled`, `blocklisted`. | | lists | number\[\] | | List of list IDs to subscribe to. | | attribs | JSON | | Optional JSON object attributes for the subscriber that can be used in message templates. Example `{"location": "Somewhere"}` | | preconfirm_subscriptions | bool | | If true, subscriptions are marked as confirmed and no opt-in emails are sent for double opt-in lists. | ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers' -H 'Content-Type: application/json' \ --data '{"email":"subscriber@domain.com","name":"The Subscriber","status":"enabled","lists":[1],"attribs":{"city":"Bengaluru","projects":3,"stack":{"languages":["go","python"]}}}' ``` ##### Example Response ```json { "data": { "id": 3, "created_at": "2019-07-03T12:17:29.735507+05:30", "updated_at": "2019-07-03T12:17:29.735507+05:30", "uuid": "eb420c55-4cfb-4972-92ba-c93c34ba475d", "email": "subscriber@domain.com", "name": "The Subscriber", "attribs": { "city": "Bengaluru", "projects": 3, "stack": { "languages": ["go", "python"] } }, "status": "enabled", "lists": [1] } } ``` ______________________________________________________________________ #### POST /api/subscribers/{subscribers_id}/optin Sends opt-in confirmation email to subscribers. ##### Example Request ```shell curl -u 'api_username:access_token' 'http://localhost:9000/api/subscribers/11/optin' -H 'Content-Type: application/json' \ --data {} ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### POST /api/public/subscription Create a public subscription, accepts both form encoded or JSON encoded body. ##### Parameters | Name | Type | Required | Description | | :--------- | :--------- | :------- | :-------------------------- | | email | string | Yes | Subscriber's email address. | | name | string | | Subscriber's name. | | list_uuids | string\[\] | Yes | List of list UUIDs. | ##### Example JSON Request ```shell curl 'http://localhost:9000/api/public/subscription' -H 'Content-Type: application/json' \ --data '{"email":"subscriber@domain.com","name":"The Subscriber","list_uuids": ["eb420c55-4cfb-4972-92ba-c93c34ba475d", "0c554cfb-eb42-4972-92ba-c93c34ba475d"]}' ``` ##### Example Form Request ```shell curl -u 'http://localhost:9000/api/public/subscription' \ -d 'email=subscriber@domain.com' -d 'name=The Subscriber' -d 'l=eb420c55-4cfb-4972-92ba-c93c34ba475d' -d 'l=0c554cfb-eb42-4972-92ba-c93c34ba475d' ``` Note: For form request, use `l` for multiple lists instead of `lists`. ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### PUT /api/subscribers/lists Modify subscriber list memberships. ##### Parameters | Name | Type | Required | Description | | :-------------- | :--------- | :----------------- | :---------------------------------------------------------------- | | ids | number\[\] | Yes | Array of user IDs to be modified. | | action | string | Yes | Action to be applied: `add`, `remove`, or `unsubscribe`. | | target_list_ids | number\[\] | Yes | Array of list IDs to be modified. | | status | string | Required for `add` | Subscriber status: `confirmed`, `unconfirmed`, or `unsubscribed`. | ##### Example Request ```shell curl -u 'api_username:access_token' -X PUT 'http://localhost:9000/api/subscribers/lists' \ -H 'Content-Type: application/json' \ --data-raw '{"ids": [1, 2, 3], "action": "add", "target_list_ids": [4, 5, 6], "status": "confirmed"}' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### PUT /api/subscribers/{subscriber_id} Update a specific subscriber. > Refer to parameters from [POST /api/subscribers](#post-apisubscribers). Note: All parameters must be set, if not, the subscriber will be removed from all previously assigned lists. ______________________________________________________________________ #### PUT /api/subscribers/{subscriber_id}/blocklist Blocklist a specific subscriber. ##### Parameters | Name | Type | Required | Description | | :------------ | :----- | :------- | :--------------- | | subscriber_id | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' -X PUT 'http://localhost:9000/api/subscribers/9/blocklist' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### PUT /api/subscribers/blocklist Blocklist multiple subscriber. ##### Parameters | Name | Type | Required | Description | | :--- | :----- | :------- | :--------------- | | ids | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' -X PUT 'http://localhost:8080/api/subscribers/blocklist' -H 'Content-Type: application/json' --data-raw '{"ids":[2,1]}' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### PUT /api/subscribers/query/blocklist Blocklist subscribers based on SQL expression. > Refer to the [querying and segmentation](../querying-and-segmentation.md#querying-and-segmenting-subscribers) section for more information on how to query subscribers with SQL expressions. ##### Parameters | Name | Type | Required | Description | | :------- | :------- | :------- | :------------------------------------------- | | query | string | Yes | SQL expression to filter subscribers with. | | list_ids | []number | No | Optional list IDs to limit the filtering to. | ##### Example Request ```shell curl -u 'api_username:access_token' -X POST 'http://localhost:9000/api/subscribers/query/blocklist' \ -H 'Content-Type: application/json' \ --data-raw '{"query":"subscribers.name LIKE \'John Doe\' AND subscribers.attribs->>'\''city'\'' = '\''Bengaluru'\''"}' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/subscribers/{subscriber_id} Delete a specific subscriber. ##### Parameters | Name | Type | Required | Description | | :------------ | :----- | :------- | :--------------- | | subscriber_id | Number | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/subscribers/9' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/subscribers/{subscriber_id}/bounces Delete a subscriber's bounce records ##### Parameters | Name | Type | Required | Description | | :--- | :------------ | :------- | :--------------- | | id | subscriber_id | Yes | Subscriber's ID. | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/subscribers/9/bounces' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### DELETE /api/subscribers Delete one or more subscribers. ##### Parameters | Name | Type | Required | Description | | :--- | :--------- | :------- | :------------------------- | | id | number\[\] | Yes | Array of subscriber's IDs. | ##### Example Request ```shell curl -u 'api_username:access_token' -X DELETE 'http://localhost:9000/api/subscribers?id=10&id=11' ``` ##### Example Response ```json { "data": true } ``` ______________________________________________________________________ #### POST /api/subscribers/query/delete Delete subscribers based on SQL expression. ##### Parameters | Name | Type | Required | Description | | :------- | :------- | :------- | :----------------------------------------------------------------- | | query | string | No | SQL expression to filter subscribers with. | | list_ids | []number | No | Optional list IDs to limit the filtering to. | | all | bool | No | When set to `true`, ignores any query and deletes all subscribers. | ##### Example Request ```shell curl -u 'api_username:access_token' -X POST 'http://localhost:9000/api/subscribers/query/delete' \ -H 'Content-Type: application/json' \ --data-raw '{"query":"subscribers.name LIKE \'John Doe\' AND subscribers.attribs->>'\''city'\'' = '\''Bengaluru'\''"}' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/templates.md ================================================ # API / Templates | Method | Endpoint | Description | |:-------|:------------------------------------------------------------------------------|:-------------------------------| | GET | [/api/templates](#get-apitemplates) | Retrieve all templates | | GET | [/api/templates/{template_id}](#get-apitemplates-template_id) | Retrieve a template | | GET | [/api/templates/{template_id}/preview](#get-apitemplates-template_id-preview) | Retrieve template HTML preview | | POST | [/api/templates](#post-apitemplates) | Create a template | | POST | /api/templates/preview | Render and preview a template | | PUT | [/api/templates/{template_id}](#put-apitemplatestemplate_id) | Update a template | | PUT | [/api/templates/{template_id}/default](#put-apitemplates-template_id-default) | Set default template | | DELETE | [/api/templates/{template_id}](#delete-apitemplates-template_id) | Delete a template | ______________________________________________________________________ #### GET /api/templates Retrieve all templates. ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/templates' ``` ##### Example Response ```json { "data": [ { "id": 1, "created_at": "2020-03-14T17:36:41.288578+01:00", "updated_at": "2020-03-14T17:36:41.288578+01:00", "name": "Default template", "body": "{{ template \"content\" . }}", "body_source": null, "type": "campaign", "is_default": true } ] } ``` ______________________________________________________________________ #### GET /api/templates/{template_id} Retrieve a specific template. ##### Parameters | Name | Type | Required | Description | |:------------|:----------|:---------|:-------------------------------| | template_id | number | Yes | ID of the template to retrieve | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/templates/1' ``` ##### Example Response ```json { "data": { "id": 1, "created_at": "2020-03-14T17:36:41.288578+01:00", "updated_at": "2020-03-14T17:36:41.288578+01:00", "name": "Default template", "body": "{{ template \"content\" . }}", "body_source": null, "type": "campaign", "is_default": true } } ``` ______________________________________________________________________ #### GET /api/templates/{template_id}/preview Retrieve the HTML preview of a template. ##### Parameters | Name | Type | Required | Description | |:------------|:----------|:---------|:------------------------------| | template_id | number | Yes | ID of the template to preview | ##### Example Request ```shell curl -u "api_user:token" -X GET 'http://localhost:9000/api/templates/1/preview' ``` ##### Example Response ```htmlHi there
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis et elit ac elit sollicitudin condimentum non a magna. Sed tempor mauris in facilisis vehicula. Aenean nisl urna, accumsan ac tincidunt vitae, interdum cursus massa. Interdum et malesuada fames ac ante ipsum primis in faucibus. Aliquam varius turpis et turpis lacinia placerat. Aenean id ligula a orci lacinia blandit at eu felis. Phasellus vel lobortis lacus. Suspendisse leo elit, luctus sed erat ut, venenatis fermentum ipsum. Donec bibendum neque quis.
Sub heading
Nam luctus dui non placerat mattis. Morbi non accumsan orci, vel interdum urna. Duis faucibus id nunc ut euismod. Curabitur et eros id erat feugiat fringilla in eget neque. Aliquam accumsan cursus eros sed faucibus.
Here is a link to listmonk.
``` ______________________________________________________________________ #### POST /api/templates Create a template. ##### Parameters | Name | Type | Required | Description | |:------------|:-------|:---------|:------------------------------------------------------------------------------| | name | string | Yes | Name of the template | | type | string | Yes | Type of the template (`campaign`, `campaign_visual`, or `tx`) | | subject | string | | Subject line for the template (only for `tx`) | | body_source | string | | If type is `campaign_visual`, the JSON source for the email-builder tempalate | | body | string | Yes | HTML body of the template | ##### Example Request ```shell curl -u "api_user:token" -X POST 'http://localhost:9000/api/templates' \ -H 'Content-Type: application/json' \ -d '{ "name": "New template", "type": "campaign", "subject": "Your Weekly Newsletter", "body": "Header
Content goes here
" }' ``` ##### Example Response ```json { "data": [ { "id": 1, "created_at": "2020-03-14T17:36:41.288578+01:00", "updated_at": "2020-03-14T17:36:41.288578+01:00", "name": "Default template", "body": "{{ template \"content\" . }}", "body_source": null, "type": "campaign", "is_default": true } ] } ``` ______________________________________________________________________ #### PUT /api/templates/{template_id} Update a template. > Refer to parameters from [POST /api/templates](#post-apitemplates) ______________________________________________________________________ #### PUT /api/templates/{template_id}/default Set a template as the default. ##### Parameters | Name | Type | Required | Description | |:------------|:----------|:---------|:-------------------------------------| | template_id | number | Yes | ID of the template to set as default | ##### Example Request ```shell curl -u "api_user:token" -X PUT 'http://localhost:9000/api/templates/1/default' ``` ##### Example Response ```json { "data": { "id": 1, "created_at": "2020-03-14T17:36:41.288578+01:00", "updated_at": "2020-03-14T17:36:41.288578+01:00", "name": "Default template", "body": "{{ template \"content\" . }}", "body_source": null, "type": "campaign", "is_default": true } } ``` ______________________________________________________________________ #### DELETE /api/templates/{template_id} Delete a template. ##### Parameters | Name | Type | Required | Description | |:------------|:----------|:---------|:-----------------------------| | template_id | number | Yes | ID of the template to delete | ##### Example Request ```shell curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/templates/35' ``` ##### Example Response ```json { "data": true } ``` ================================================ FILE: docs/docs/content/apis/transactional.md ================================================ # API / Transactional | Method | Endpoint | Description | | :----- | :------- | :-------------------------- | | POST | /api/tx | Send transactional messages | ______________________________________________________________________ #### POST /api/tx Allows sending transactional messages to one or more subscribers via a preconfigured transactional template. ##### Parameters | Name | Type | Required | Description | | :---------------- | :--------- | :------- | :------------------------------------------------------------------------- | | subscriber_email | string | | Email of the subscriber. Can substitute with `subscriber_id`. | | subscriber_id | number | | Subscriber's ID can substitute with `subscriber_email`. | | subscriber_emails | string\[\] | | Multiple subscriber emails as alternative to `subscriber_email`. | | subscriber_ids | number\[\] | | Multiple subscriber IDs as an alternative to `subscriber_id`. | | subscriber_mode | string | | Subscriber lookup mode: `default`, `fallback`, or `external` | | template_id | number | Yes | ID of the transactional template to be used for the message. | | from_email | string | | Optional sender email. | | subject | string | | Optional subject. If empty, the subject defined on the template is used | | data | JSON | | Optional nested JSON map. Available in the template as `{{ .Tx.Data.* }}`. | | headers | JSON\[\] | | Optional array of email headers. | | messenger | string | | Messenger to send the message. Default is `email`. | | content_type | string | | Email format options include `html`, `markdown`, and `plain`. | | altbody | string | | Optional alternate plaintext body for multipart HTML emails. | ##### Subscriber modes The `subscriber_mode` parameter controls how the recipients (subscribers or non-subscriber recipients) are resolved. | Mode | Description | | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `default` | Recipients must exist as subscribers in the database. Pass either `subscriber_emails` or `subscriber_ids`. | | `fallback` | Only accepts `subscriber_emails` and looks up subscribers in the database. If not found, sends the message to the e-mail anyway. In the template, apart from `{{ .Subscriber.Email }}`, other subscriber fields such as `.Name`. will be empty. Use `{{ Tx.Data.* }}` instead. | | `external` | Sends to the given `subscriber_emails` without subscriber lookup in the database. In the template, apart from `{{ .Subscriber.Email }}`, other subscriber fields such as `.Name`. will be empty. Use `{{ Tx.Data.* }}` instead. | ##### Example ```shell curl -u "api_user:token" "http://localhost:9000/api/tx" -X POST \ -H 'Content-Type: application/json; charset=utf-8' \ --data-binary @- << EOF { "subscriber_email": "user@test.com", "template_id": 2, "data": {"order_id": "1234", "date": "2022-07-30", "items": [1, 2, 3]}, "content_type": "html" } EOF ``` ##### Example response ```json { "data": true } ``` ##### Example with external mode Send to arbitrary email addresses without requiring them to be subscribers: ```shell curl -u "api_user:token" "http://localhost:9000/api/tx" -X POST \ -H 'Content-Type: application/json; charset=utf-8' \ --data-binary @- << EOF { "subscriber_mode": "external", "subscriber_emails": ["recipient@example.com"], "template_id": 2, "data": {"name": "John", "order_id": "1234"}, "content_type": "html" } EOF ``` In the template, use `{{ .Tx.Data.name }}`, `{{ .Tx.Data.order_id }}`, etc. to access the data. ______________________________________________________________________ #### File Attachments To include file attachments in a transactional message, use the `multipart/form-data` Content-Type. Use `data` param for the parameters described above as a JSON object. Include any number of attachments via the `file` param. ```shell curl -u "api_user:token" "http://localhost:9000/api/tx" -X POST \ -F 'data=\"{ \"subscriber_email\": \"user@test.com\", \"template_id\": 4 }"' \ -F 'file=@"/path/to/attachment.pdf"' \ -F 'file=@"/path/to/attachment2.pdf"' ``` ================================================ FILE: docs/docs/content/archives.md ================================================ # Archives A global public archive is maintained on the public web interface. It can be enabled under Settings -> Settings -> General -> Enable public mailing list archive. To make a campaign available in the public archive (provided it has been enabled in the settings as described above), enable the option 'Publish to public archive' under Campaigns -> Create new -> Archive. When using template variables that depend on subscriber data (such as any template variable referencing `.Subscriber`), such data must be supplied as 'Campaign metadata', which is a JSON object that will be used in place of `.Subscriber` when rendering the archive template and content. When individual subscriber tracking is enabled, TrackLink requires that a UUID of an existing user is provided as part of the campaign metadata. Any clicks on a TrackLink from the archived campaign will be counted towards that subscriber. As an example: ```json { "UUID": "5a837423-a186-5623-9a87-82691cbe3631", "email": "example@example.com", "name": "Reader", "attribs": {} } ```  ================================================ FILE: docs/docs/content/bounces.md ================================================ # Bounce processing Enable bounce processing in Settings -> Bounces. POP3 bounce scanning and APIs only become available once the setting is enabled. ## POP3 bounce mailbox Configure the bounce mailbox in Settings -> Bounces. Either the "From" e-mail that is set on a campaign (or in settings) should have a POP3 mailbox behind it to receive bounce e-mails, or you should configure a dedicated POP3 mailbox and add that address as the `Return-Path` (envelope sender) header in Settings -> SMTP -> Custom headers box. For example: ``` [ {"Return-Path": "your-bounce-inbox@site.com"} ] ``` Some mail servers may also return the bounce to the `Reply-To` address, which can also be added to the header settings. ### Bounce classification listmonk applies a series of heuristics looking for keywords in the bounced mail body to guess if it is a 'soft' bounce or a 'hard' bounce. For instance, 4.x.x and 5.x.x error status codes, common strings such as "mailbox not found" etc. If none of the heuristics match, then the bounce mail is considered to be 'soft' by default. ## Webhook API The bounce webhook API can be used to record bounce events with custom scripting. This could be by reading a mailbox, a database, or mail server logs. | Method | Endpoint | Description | | ------ | ---------------- | ---------------------- | | `POST` | /webhooks/bounce | Record a bounce event. | | Name | Type | Required | Description | | --------------- | ------ | -------- | ------------------------------------------------------------------------------------ | | subscriber_uuid | string | | The UUID of the subscriber. Either this or `email` is required. | | email | string | | The e-mail of the subscriber. Either this or `subscriber_uuid` is required. | | campaign_uuid | string | | UUID of the campaign for which the bounce happened. | | source | string | Yes | A string indicating the source, eg: `api`, `my_script` etc. | | type | string | Yes | `hard` or `soft` bounce. Currently, this has no effect on how the bounce is treated. | | meta | string | | An optional escaped JSON string with arbitrary metadata about the bounce event. | ```shell curl -u 'api_username:access_token' -X POST 'http://localhost:9000/webhooks/bounce' \ -H "Content-Type: application/json" \ --data '{"email": "user1@mail.com", "campaign_uuid": "9f86b50d-5711-41c8-ab03-bc91c43d711b", "source": "api", "type": "hard", "meta": "{\"additional\": \"info\"}}' ``` ## External webhooks listmonk supports receiving bounce webhook events from the following SMTP providers. | Endpoint | Description | More info | | :------------------------------------------------------------ | :------------------------------------- | :-------------------------------------------------------------------------------------------------------------------- | | `https://listmonk.yoursite.com/webhooks/service/ses` | Amazon (AWS) SES | See below | | `https://listmonk.yoursite.com/webhooks/service/sendgrid` | Sendgrid / Twilio Signed event webhook | [More info](https://docs.sendgrid.com/for-developers/tracking-events/getting-started-event-webhook-security-features) | | `https://listmonk.yoursite.com/webhooks/service/postmark` | Postmark webhook | [More info](https://postmarkapp.com/developer/webhooks/webhooks-overview) | | `https://listmonk.yoursite.com/webhooks/service/forwardemail` | Forward Email webhook | [More info](https://forwardemail.net/en/faq#do-you-support-bounce-webhooks) | ## Amazon Simple Email Service (SES) If using SES as your SMTP provider, automatic bounce processing is the recommended way to maintain your [sender reputation](https://docs.aws.amazon.com/ses/latest/dg/monitor-sender-reputation.html). The settings below are based on Amazon's [recommendations](https://docs.aws.amazon.com/ses/latest/dg/send-email-concepts-deliverability.html). Please note that your sending domain must be verified in SES before proceeding. 1. In listmonk settings, go to the "Bounces" tab and configure the following: - Enable bounce processing: `Enabled` - Soft: - Bounce count: `2` - Action: `None` - Hard: - Bounce count: `1` - Action: `Blocklist` - Complaint: - Bounce count: `1` - Action: `Blocklist` - Enable bounce webhooks: `Enabled` - Enable SES: `Enabled` 2. In the AWS console, go to [Simple Notification Service](https://console.aws.amazon.com/sns/) and create a new topic with the following settings: - Type: `Standard` - Name: `ses-bounces` (or any other name) 3. Create a new subscription to that topic with the following settings: - Protocol: `HTTPS` - Endpoint: `https://listmonk.yoursite.com/webhooks/service/ses` - Enable raw message delivery: `Disabled` (unchecked) 4. SES will then make a request to your listmonk instance to confirm the subscription. After a page refresh, the subscription should have a status of "Confirmed". If not, your endpoint may be incorrect or not publicly accessible. 5. In the AWS console, go to [Simple Email Service](https://console.aws.amazon.com/ses/) and click "Identities" in the left sidebar. 6. Click your domain and go to the "Notifications" tab. 7. Next to "Feedback notifications", click "Edit". 8. For both "Bounce feedback" and "Complaint feedback", use the following settings: - SNS topic: `ses-bounces` (or whatever you named it) - Include original email headers: `Enabled` (checked) 9. Repeat steps 6-8 for any `Email address` identities you send from using listmonk 10. Bounce processing should now be working. You can test it with [SES simulator addresses](https://docs.aws.amazon.com/ses/latest/dg/send-an-email-from-console.html#send-email-simulator). Add them as subscribers, send them campaign previews, and ensure that the appropriate action was taken after the configured bounce count was reached. - Soft bounce: `ooto@simulator.amazonses.com` - Hard bounce: `bounce@simulator.amazonses.com` - Complaint: `complaint@simulator.amazonses.com` 11. You can optionally [disable email feedback forwarding](https://docs.aws.amazon.com/ses/latest/dg/monitor-sending-activity-using-notifications-email.html#monitor-sending-activity-using-notifications-email-disabling). ## Exporting bounces Bounces can be exported via the JSON API: ```shell curl -u 'username:passsword' 'http://localhost:9000/api/bounces' ``` Or by querying the database directly: ```sql SELECT bounces.created_at, bounces.subscriber_id, subscribers.uuid AS subscriber_uuid, subscribers.email AS email FROM bounces LEFT JOIN subscribers ON (subscribers.id = bounces.subscriber_id) ORDER BY bounces.created_at DESC LIMIT 1000; ``` ================================================ FILE: docs/docs/content/concepts.md ================================================ # Concepts ## Subscriber A subscriber is a recipient identified by an e-mail address and name. Subscribers receive e-mails that are sent from listmonk. A subscriber can be added to any number of lists. Subscribers who are not a part of any lists are considered *orphan* records. ### Attributes Attributes are arbitrary properties attached to a subscriber in addition to their e-mail and name. They are represented as a JSON map. It is not necessary for all subscribers to have the same attributes. Subscribers can be [queried and segmented](querying-and-segmentation.md) into lists based on their attributes, and the attributes can be inserted into the e-mails sent to them. For example: ```json { "city": "Bengaluru", "likes_tea": true, "spoken_languages": ["English", "Malayalam"], "projects": 3, "stack": { "frameworks": ["echo", "go"], "languages": ["go", "python"], "preferred_language": "go" } } ``` ### Subscription statuses A subscriber can be added to one or more lists, and each such relationship can have one of these statuses. | Status | Description | |----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `unconfirmed` | The subscriber was added to the list directly without their explicit confirmation. Nonetheless, the subscriber will receive campaign messages sent to single opt-in campaigns. | | `confirmed` | The subscriber confirmed their subscription by clicking on 'accept' in the confirmation e-mail. Only confirmed subscribers in opt-in lists will receive campaign messages send to the list. | | `unsubscribed` | The subscriber is unsubscribed from the list and will not receive any campaign messages sent to the list. | ### Segmentation Segmentation is the process of filtering a large list of subscribers into a smaller group based on arbitrary conditions, primarily based on their attributes. For instance, if an e-mail needs to be sent subscribers who live in a particular city, given their city is described in their attributes, it's possible to quickly filter them out into a new list and e-mail them. [Learn more](querying-and-segmentation.md). ## List A list (or a _mailing list_) is a collection of subscribers grouped under a name, for instance, _clients_. Lists are used to organise subscribers and send e-mails to specific groups. A list can be single opt-in or double opt-in. Subscribers added to double opt-in lists have to explicitly accept the subscription by clicking on the confirmation e-mail they receive. Until then, they do not receive campaign messages. ## Campaign A campaign is an e-mail (or any other kind of messages) that is sent to one or more lists. ## Transactional message A transactional message is an arbitrary message sent to a subscriber using the transactional message API. For example a welcome e-mail on signing up to a service; an order confirmation e-mail on purchasing an item; a password reset e-mail when a user initiates an online account recovery process. ## Template A template is a re-usable HTML design that can be used across campaigns and when sending arbitrary transactional messages. Most commonly, templates have standard header and footer areas with logos and branding elements, where campaign content is inserted in the middle. listmonk supports [Go template](https://gowebexamples.com/templates/) expressions that lets you create powerful, dynamic HTML templates. [Learn more](templating.md). ## Messenger listmonk supports multiple custom messaging backends in additional to the default SMTP e-mail backend, enabling not just e-mail campaigns, but arbitrary message campaigns such as SMS, FCM notifications etc. A *Messenger* is a web service that accepts a campaign message pushed to it as a JSON request, which the service can in turn broadcast as SMS, FCM etc. [Learn more](messengers.md). ## Tracking pixel The tracking pixel is a tiny, invisible image that is inserted into an e-mail body to track e-mail views. This allows measuring the read rate of e-mails. While this is exceedingly common in e-mail campaigns, it carries privacy implications and should be used in compliance with rules and regulations such as GDPR. It is possible to track reads anonymously without associating an e-mail read to a subscriber. ## Click tracking It is possible to track the clicks on every link that is sent in an e-mail. This allows measuring the clickthrough rates of links in e-mails. While this is exceedingly common in e-mail campaigns, it carries privacy implications and should be used in compliance with rules and regulations such as GDPR. It is possible to track link clicks anonymously without associating an e-mail read to a subscriber. ## Bounce A bounce occurs when an e-mail that is sent to a recipient "bounces" back for one of many reasons including the recipient address being invalid, their mailbox being full, or the recipient's e-mail service provider marking the e-mail as spam. listmonk can automatically process such bounce e-mails that land in a configured POP mailbox, or via APIs of SMTP e-mail providers such as AWS SES and Sengrid. Based on settings, subscribers returning bounced e-mails can either be blocklisted or deleted automatically. [Learn more](bounces.md). ================================================ FILE: docs/docs/content/configuration.md ================================================ # Configuration ### TOML Configuration file One or more TOML files can be read by passing `--config config.toml` multiple times. Apart from a few low level configuration variables and the database configuration, all other settings can be managed from the `Settings` dashboard on the admin UI. To generate a new sample configuration file, run `listmonk --new-config` ### Environment variables Variables in config.toml can also be provided as environment variables prefixed by `LISTMONK_` with periods replaced by `__` (double underscore). To start listmonk purely with environment variables without a configuration file, set the environment variables and pass the config flag as `--config=""`. Example: | **Environment variable** | Example value | | ------------------------------ | -------------- | | `LISTMONK_app__address` | "0.0.0.0:9000" | | `LISTMONK_db__host` | db | | `LISTMONK_db__port` | 9432 | | `LISTMONK_db__user` | listmonk | | `LISTMONK_db__password` | listmonk | | `LISTMONK_db__database` | listmonk | | `LISTMONK_db__ssl_mode` | disable | ### Customizing system templates See [system templates](templating.md#system-templates). ### HTTP routes When configuring auth proxies and web application firewalls, use this table. #### Private admin endpoints. | Methods | Route | Description | | ------- | ------------------ | ----------------------- | | `*` | `/api/*` | Admin APIs | | `GET` | `/admin/*` | Admin UI and HTML pages | | `POST` | `/webhooks/bounce` | Admin bounce webhook | #### Public endpoints to expose to the internet. | Methods | Route | Description | | ----------- | --------------------- | --------------------------------------------- | | `GET, POST` | `/subscription/*` | HTML subscription pages | | `GET, ` | `/link/*` | Tracked link redirection | | `GET` | `/campaign/*` | Pixel tracking image | | `GET` | `/public/*` | Static files for HTML subscription pages | | `POST` | `/webhooks/service/*` | Bounce webhook endpoints for AWS and Sendgrid | | `GET` | `/uploads/*` | The file upload path configured in media settings | ## Media uploads #### Using filesystem When configuring `docker` volume mounts for using filesystem media uploads, you can follow either of two approaches. [The second option may be necessary if](https://github.com/knadh/listmonk/issues/1169#issuecomment-1674475945) your setup requires you to use `sudo` for docker commands. After making any changes you will need to run `sudo docker compose stop ; sudo docker compose up`. And under `https://listmonk.mysite.com/admin/settings` you put `/listmonk/uploads`. #### Using volumes Using `docker volumes`, you can specify the name of volume and destination for the files to be uploaded inside the container. ```yml app: volumes: - type: volume source: listmonk-uploads target: /listmonk/uploads volumes: listmonk-uploads: ``` !!! note This volume is managed by `docker` itself, and you can see find the host path with `docker volume inspect listmonk_listmonk-uploads`. #### Using bind mounts ```yml app: volumes: - ./path/on/your/host/:/path/inside/container ``` Eg: ```yml app: volumes: - ./data/uploads:/listmonk/uploads ``` The files will be available inside `/data/uploads` directory on the host machine. To use the default `uploads` folder: ```yml app: volumes: - ./uploads:/listmonk/uploads ``` ## Logs ### Docker https://docs.docker.com/engine/reference/commandline/logs/ ``` sudo docker logs -f sudo docker logs listmonk_app -t sudo docker logs listmonk_db -t sudo docker logs --help ``` Container info: `sudo docker inspect listmonk_listmonk` Docker logs to `/dev/stdout` and `/dev/stderr`. The logs are collected by the docker daemon and stored in your node's host path (by default). The same can be configured (/etc/docker/daemon.json) in your docker daemon settings to setup other logging drivers, logrotate policy and more, which you can read about [here](https://docs.docker.com/config/containers/logging/configure/). ### Binary listmonk logs to `stdout`, which is usually not saved to any file. To save listmonk logs to a file use `./listmonk > listmonk.log`. Settings -> Logs in admin shows the last 1000 lines of the standard log output but gets erased when listmonk is restarted. For the [service file](https://github.com/knadh/listmonk/blob/master/listmonk%40.service), you can use `ExecStart=/bin/bash -ce "exec /usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static >>/etc/listmonk/listmonk.log 2>&1"` to create a log file that persists after restarts. [More info](https://github.com/knadh/listmonk/issues/1462#issuecomment-1868501606). ## Time zone To change listmonk's time zone (logs, etc.) edit `docker-compose.yml`: ``` environment: - TZ=Etc/UTC ``` with any Timezone listed [here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). Then run `sudo docker-compose stop ; sudo docker-compose up` after making changes. ## SMTP ### Retries The `Settings -> SMTP -> Retries` denotes the number of times a message that fails at the moment of sending is retried silently using different connections from the SMTP pool. The messages that fail even after retries are the ones that are logged as errors and ignored. ## SMTP ports Some server hosts block outgoing SMTP ports (25, 465). You may have to contact your host to unblock them before being able to send e-mails. Eg: [Hetzner](https://docs.hetzner.com/cloud/servers/faq/#why-can-i-not-send-any-mails-from-my-server). ## Performance ### Batch size The batch size parameter is useful when working with very large lists with millions of subscribers for maximising throughput. It is the number of subscribers that are fetched from the database sequentially in a single cycle (~5 seconds) when a campaign is running. Increasing the batch size uses more memory, but reduces the round trip to the database. ================================================ FILE: docs/docs/content/developer-setup.md ================================================ # Developer setup The app has two distinct components, the Go backend and the VueJS frontend. In the dev environment, both are run independently. ### Pre-requisites - `go` - `nodejs` (if you are working on the frontend) and `yarn` - Postgres database. If there is no local installation, the demo docker DB can be used for development (`docker compose up demo-db`) ### First time setup `git clone https://github.com/knadh/listmonk.git`. The project uses go.mod, so it's best to clone it outside the Go src path. 1. Copy `config.toml.sample` as `config.toml` and add your config. 2. `make dist` to build the listmonk binary. Once the binary is built, run `./listmonk --install` to run the DB setup. For subsequent dev runs, use `make run`. > [mailhog](https://github.com/mailhog/MailHog) is an excellent standalone mock SMTP server (with a UI) for testing and dev. ### Running the dev environment You can run your dev environment locally or inside containers. After setting up the dev environment, you can visit `http://localhost:8080`. 1. Locally - Run `make run` to start the listmonk dev server on `:9000`. - Run `make run-frontend` to start the Vue frontend in dev mode using yarn on `:8080`. All `/api/*` calls are proxied to the app running on `:9000`. Refer to the [frontend README](https://github.com/knadh/listmonk/blob/master/frontend/README.md) for an overview on how the frontend is structured. 2. Inside containers (Using Makefile) - Run `make init-dev-docker` to setup container for db. - Run `make dev-docker` to setup docker container suite. - Run `make rm-dev-docker` to clean up docker container suite. 3. Inside containers (Using devcontainer) - Open repo in vscode, open command palette, and select "Dev Containers: Rebuild and Reopen in Container". It will set up db, and start frontend/backend for you. # Production build Run `make dist` to build the Go binary, build the Javascript frontend, and embed the static assets producing a single self-contained binary, `listmonk` ================================================ FILE: docs/docs/content/external-integration.md ================================================ # Integrating with external systems In many environments, a mailing list manager's subscriber database is not run independently but as a part of an existing customer database or a CRM. There are multiple ways of keeping listmonk in sync with external systems. ## Using APIs The [subscriber APIs](apis/subscribers.md) offers several APIs to manipulate the subscribers database, like addition, updation, and deletion. For bulk synchronisation, a CSV can be generated (and optionally zipped) and posted to the import API. ## Interacting directly with the DB listmonk uses tables with simple schemas to represent subscribers (`subscribers`), lists (`lists`), and subscriptions (`subscriber_lists`). It is easy to add, update, and delete subscriber information directly with the database tables for advanced usecases. See the [table schemas](https://github.com/knadh/listmonk/blob/master/schema.sql) for more information. ================================================ FILE: docs/docs/content/i18n.md ================================================ # Internationalization (i18n) listmonk comes available in multiple languages thanks to language packs contributed by volunteers. A language pack is a JSON file with a map of keys and corresponding translations. The bundled languages can be [viewed here](https://github.com/knadh/listmonk/tree/master/i18n). ## Additional language packs These additional language packs can be downloaded and passed to listmonk with the `--i18n-dir` flag as described in the next section. | Language | Description | |------------------|--------------------------------------| | [Deutsch (formal)](https://raw.githubusercontent.com/SvenPe/listmonk/4bbb2e5ebb2314b754cb2318f4f6683a0f854d43/i18n/de.json) | German language with formal pronouns | ## Customizing languages To customize an existing language or to load a new language, put one or more `.json` language files in a directory, and pass the directory path to listmonk with the`--i18n-dir=/path/to/dir` flag. ## Contributing a new language ### Using the basic editor - Visit [https://listmonk.app/i18n](https://listmonk.app/i18n) - Click on `Createa new language`, or to make changes to an existing language, use `Load language`. - Translate the text in the text fields on the UI. - Once done, use the `Download raw JSON` to download the language file. - Send a pull request to add the file to the [i18n directory on the GitHub repo](https://github.com/knadh/listmonk/tree/master/i18n). ### Using InLang (external service) [](https://inlang.com/editor/github.com/knadh/listmonk?ref=badge) - Visit [https://inlang.com/editor/github.com/knadh/listmonk](https://inlang.com/editor/github.com/knadh/listmonk) - To make changes and push them, you need to log in to GitHub using OAuth and fork the project from the UI. - Translate the text in the input fields on the UI. You can use the filters to see only the necessary translations. - Once you're done, push the changes from the UI and click on "Open a pull request." This will take you to GitHub, where you can write a PR message. ================================================ FILE: docs/docs/content/index.md ================================================ # Introduction [](https://listmonk.app) listmonk is a self-hosted, high performance one-way mailing list and newsletter manager. It comes as a standalone binary and the only dependency is a Postgres database. [](https://listmonk.app) ## Developers listmonk is a free and open source software licensed under AGPLv3. If you are interested in contributing, check out the [GitHub repository](https://github.com/knadh/listmonk) and refer to the [developer setup](developer-setup.md). The backend is written in Go and the frontend is Vue with Buefy for UI. ================================================ FILE: docs/docs/content/installation.md ================================================ # Installation listmonk is a simple binary application that requires a Postgres database instance to run. The binary can be downloaded and run manually, or it can be run as a container with Docker compose. ## Binary 1. Download the [latest release](https://github.com/knadh/listmonk/releases) and extract the listmonk binary. `amd64` is the main one. It works for Intel and x86 CPUs. 1. `./listmonk --new-config` to generate config.toml. Edit the file. 1. `./listmonk --install` to install the tables in the Postgres DB (⩾ 12). 1. Run `./listmonk` and visit `http://localhost:9000` to create the Super Admin user and login. !!! Tip To set the Super Admin username and password during installation, set the environment variables: `LISTMONK_ADMIN_USER=myuser LISTMONK_ADMIN_PASSWORD=xxxxx ./listmonk --install` ## Docker The latest image is available on DockerHub at `listmonk/listmonk:latest` The recommended method is to download the [docker-compose.yml](https://github.com/knadh/listmonk/blob/master/docker-compose.yml) file, customize it for your environment and then to simply run `docker compose up -d`. ```shell # Download the compose file to the current directory. curl -LO https://github.com/knadh/listmonk/raw/master/docker-compose.yml # Run the services in the background. docker compose up -d ``` Then, visit `http://localhost:9000` to create the Super Admin user and login. !!! Tip To set the Super Admin username and password during setup, set the environment variables (only the first time): `LISTMONK_ADMIN_USER=myuser LISTMONK_ADMIN_PASSWORD=xxxxx docker compose up -d` ### Mounting a custom config.toml The docker-compose file includes all necessary listmonk configuration as environment variables, `LISTMONK_*`. If you would like to remove those and mount a config.toml instead: #### 1. Save the config.toml file on the host ```toml [app] address = "0.0.0.0:9000" # Database. [db] host = "listmonk_db" # Postgres container name in the compose file. port = 5432 user = "listmonk" password = "listmonk" database = "listmonk" ssl_mode = "disable" max_open = 25 max_idle = 25 max_lifetime = "300s" ``` #### 2. Mount the config file in docker-compose.yml ```yaml app: ... volumes: - /path/on/your/host/config.toml:/listmonk/config.toml ``` #### 3. Change the `--config ''` flags in the `command:` section to point to the path ```yaml command: [sh, -c, "./listmonk --install --idempotent --yes --config /listmonk/config.toml && ./listmonk --upgrade --yes --config /listmonk/config.toml && ./listmonk --config /listmonk/config.toml"] ``` ----------- ## Nightly !!! Warning Nightly releases are untested and may have bugs. Use at your own risk. Always take a backup of your Postgres database before using a nightly release. A nightly build is automatically published with the latest changes merged to the repository. If you want to access the latest changes without waiting for versioned releases, you can obtain the nightly builds and follow the same instructions above. - **Docker**: `listmonk/listmonk:nightly` (use this as the image name in the docker-compose file) - **Binary**: [Download nightly release](https://github.com/knadh/listmonk/releases/nightly) ## Compiling from source To compile the latest unreleased version (`master` branch): 1. Make sure `go`, `nodejs`, and `yarn` are installed on your system. 2. `git clone git@github.com:knadh/listmonk.git` 3. `cd listmonk && make dist`. This will generate the `listmonk` binary. ## Helm chart for Kubernetes    A helm chart for easily installing listmonk on a kubernetes cluster is made available by community [here](https://github.com/th0th/helm-charts/tree/main/charts/listmonk). In order to use the helm chart, you can configure `values.yaml` according to your needs, and then run the following command: ```shell $ helm upgrade \ --create-namespace \ --install listmonk listmonk \ --namespace listmonk \ --repo https://th0th.github.io/helm-charts \ --values values.yaml \ --version 0.1.0 ``` ## 3rd party hosting
## Tutorials
* [Listmonk with Forward Email for Secure Newsletter Delivery](https://forwardemail.net/en/guides/newsletter-with-listmonk)
* [Informal step-by-step on how to get started with listmonk using *Railway*](https://github.com/knadh/listmonk/issues/120#issuecomment-1421838533)
* [Step-by-step tutorial for installation and all basic functions. *Amazon EC2, SES, docker & binary*](https://gist.github.com/MaximilianKohler/e5158fcfe6de80a9069926a67afcae11)
* [Step-by-step guide on how to install and set up listmonk on *AWS Lightsail with docker* (rameerez)](https://github.com/knadh/listmonk/issues/1208)
* [Quick setup on any cloud server using *docker and caddy*](https://github.com/samyogdhital/listmonk-caddy-reverse-proxy)
* [*Binary* install on Ubuntu 22.04 as a service](https://mumaritc.hashnode.dev/how-to-install-listmonk-using-binary-on-ubuntu-2204)
* [*Binary* install on Ubuntu 18.04 as a service (Apache & Plesk)](https://devgypsy.com/post/2020-08-18-installing-listmonk-newsletter-manager/)
* [*Binary and docker* on linux (techviewleo)](https://techviewleo.com/manage-mailing-list-and-newsletter-using-listmonk/)
* [*Binary* install on your PC](https://www.youtube.com/watch?v=fAOBqgR9Yfo). Discussions of limitations: [[1](https://github.com/knadh/listmonk/issues/862#issuecomment-1307328228)][[2](https://github.com/knadh/listmonk/issues/248#issuecomment-1320806990)].
* [*Docker on Rocky Linux 8* (nginx, Let's Encrypt SSL)](https://wiki.crowncloud.net/?How_to_Install_Listmonk_with_Docker_on_Rocky_Linux_8)
* [*Docker* with nginx reverse proxy, certbot SSL, and Gmail SMTP](https://www.maketecheasier.com/create-own-newsletter-with-listmonk/)
* [Install Listmonk on Self-hosting with *Pre-Configured AMI Package at AWS* by Single Click](https://meetrix.io/articles/how-to-install-llama-2-on-aws-with-pre-configured-ami-package/)
* [*Fly.io* working example](https://gitlab.com/votelog/apps/newsletter)
================================================
FILE: docs/docs/content/maintenance/performance.md
================================================
# Performance
listmonk is built to be highly performant and can handle millions of subscribers with minimal system resources.
However, as the Postgres database grows—with a large number of subscribers, campaign views, and click records—it can significantly slow down certain aspects of the program, particularly in counting records and aggregating various statistics. For instance, loading admin pages that do these aggregations can take tens of seconds if the database has millions of subscribers.
- Aggregate counts, statistics, and charts on the landing dashboard.
- Subscriber count beside every list on the Lists page.
- Total subscriber count on the Subscribers page.
However, at that scale, viewing the exact number of subscribers or statistics every time the admin panel is accessed becomes mostly unnecessary. On installations with millions of subscribers, where the above pages do not load instantly, it is highly recommended to turn on the `Settings -> Performance -> Cache slow database queries` option.
## Slow query caching
When this option is enabled, the subscriber counts on the Lists page, the Subscribers page, and the statistics on the dashboard, etc., are no longer counted in real-time in the database. Instead, they are updated periodically and cached, resulting in a massive performance boost. The periodicity can be configured on the Settings -> Performance page using a standard crontab expression (default: `0 3 * * *`, which means 3 AM daily). Use a tool like [crontab.guru](https://crontab.guru) for easily generating a desired crontab expression.
## VACUUM-ing
Running [`VACUUM ANALYZE`](https://www.postgresql.org/docs/current/sql-vacuum.html) on large Postgres databases at regular intervals (for instance, once a week), is recommended. It reclaims disk space and improves Postgres' query performance. Do note that this is a blocking operation and all database queries can come to a stand-still on a large database while the operation is running (generally only a few seconds).
================================================
FILE: docs/docs/content/messengers.md
================================================
# Messengers
listmonk supports multiple custom messaging backends in additional to the default SMTP e-mail backend, enabling not just e-mail campaigns, but arbitrary message campaigns such as SMS, FCM notifications etc.
A *Messenger* is a web service that accepts a campaign message pushed to it as a JSON request, which the service can in turn broadcast as SMS, FCM etc. Messengers are registered in the *Settings -> Messengers* UI, and can be selected on individual campaigns.
Messengers support optional BasicAuth authentication. `Plain text` format for campaign content is ideal for messengers such as SMS and FCM.
When a campaign starts, listmonk POSTs messages in the following format to the selected messenger's endpoint. The endpoint should return a `200 OK` response in case of a successful request.
The address required to broadcast the message, for instance, a phone number or an FCM ID, is expected to be stored and relayed as [subscriber attributes](concepts.md/#attributes).
```json
{
"subject": "Welcome to listmonk",
"body": "The message body",
"content_type": "plain",
"recipients": [{
"uuid": "e44b4135-1e1d-40c5-8a30-0f9a886c2884",
"email": "anon@example.com",
"name": "Anon Doe",
"attribs": {
"phone": "123123123",
"fcm_id": "2e7e4b512e7e4b512e7e4b51",
"city": "Bengaluru"
},
"status": "enabled"
}],
"campaign": {
"uuid": "2e7e4b51-f31b-418a-a120-e41800cb689f",
"name": "Test campaign",
"tags": ["test-campaign"]
}
}
```
## Messenger implementations
Following is a list of HTTP messenger servers that connect to various backends.
| Name | Backend |
|:-------------------------------------------------------------------------------------|:-----------------|
| [listmonk-messenger](https://github.com/joeirimpan/listmonk-messenger) | AWS Pinpoint SMS |
| [listmonk-verimor-gateway](https://github.com/antandros/listmonk-verimor-gateway) | Verimor |
| [listmonk-mailersend](https://github.com/tkawczynski/listmonk-mailersend) | Mailersend |
| [listmonk-novu-messenger](https://github.com/Codepowercode/listmonk-novu-messenger) | Novu |
| [listmonk-push-messenger](https://github.com/shyamkrishna21/listmonk-push-messenger) | Google FCM |
================================================
FILE: docs/docs/content/oidc.md
================================================
## OIDC Single Sign On
Listmonk supports single sign-on with OIDC (OpenID Connect). Any standards compliant OIDC provider can be configured in Settings -> Security -> OIDC
### User auto-creation
If `Settings -> Security -> OIDC -> Auto-create users` is turned on, when users login via OIDC, an account is auto-created if an existing account is not found (based on the OIDC e-mail ID).
# Tutorials
Tutorials for configuring listmonk SSO with popular OIDC providers.
## Keycloak
Keycloak configuration for listmonk SSO integration.
### 1. Create a new client in Keycloak
In the Keycloak admin, use an existing realm, or create a new realm. Create a new client in `Clients → Create`.
- **General Settings**
- **Client type**: `OpenID Connect`
- **Client ID**: `listmonk` (or any preferred name)
- **Name**: Optional descriptive name (e.g., "listmonk SSO")
- **Capability Config**:
- **Client authentication**: On
- **Authorization**: On
- **Authentication Flow**
- **Standard Flow**: On
- **Direct Access grants**: On
- **Login Settings**:
- **Root URL**: Copy the **Redirect URL for oAuth provider** value from listmonk Admin -> Settings -> Security -> OIDC. It will look like `https://listmonk.yoursite.com/auth/oidc`
- **Valid redirect URIs**: Same as the Root URL above
- **Valid post logout redirect URIs**: *
After the client creation steps above, go to the client's `Credentials` tab and copy the `Client Secret`.
### 2. Configure Listmonk
2. In Listmonk Admin -> Settings -> Security -> OIDC.
- **Enable OIDC SSO**: Turn on
- **Provider URL**: `https://keycloak.yoursite.com/realms/{realm}` (replace `{realm}` with the chosen realm name). This URL is as of v26.3 and may differ across Keycloak versions.
- **Provider name**: Set a name to show on the listmonk login form, eg: `Login with OrgName`
- **Client ID**: Client ID set in Keycloak, eg: `listmonk`
- **Client Secret**: Client Secret copied from Keycloak
- **Auto-create users from SSO**: (Optional) Enable to automatically create users who don't exist
- **Default user role**: (Required if auto-create enabled) Select role for new users
## Authentik
Authentik configuration for listmonk SSO integration.
### 1. Create a new OIDC provider in Authentik
In the Authentik admin interface, create a new OIDC provider for listmonk.
- **Provider Settings**:
- **Name**: `listmonk` (or any preferred name)
- **Signing Key**: `authentik Self-signed Certificate`
- **Client Type**: `Confidential`
- **Client ID**: `listmonk` (or any preferred name)
- **Redirect URIs**: Copy the **Redirect URL for oAuth provider** value from listmonk Admin -> Settings -> Security -> OIDC. It will look like `https://listmonk.yoursite.com/auth/oidc`
After creating the provider, copy the **Client Secret**.
### 2. Create an application in Authentik
Create a new application and connect it to the newly created provider.
- **Application Settings**:
- **Name**: `listmonk` (or any preferred name)
- **Slug**: `listmonk` (or any preferred slug. Used in the redirect URL)
- **Provider**: Select the OIDC provider created in the previous step
### 3. Configure listmonk
In listmonk Admin → Settings → Security → OIDC:
- **Enable OIDC SSO**: Turn on
- **Provider URL**: `https://authentik.yoursite.com/application/o/{slug}/` (replace `{slug}` with the application's slug)
- **Provider Name**: Set a name to show on the login form (e.g., `Login with OrgName`)
- **Client ID**: Client ID set in Authentik (e.g., `listmonk`)
- **Client Secret**: Client Secret copied from Authentik
- **Auto-create users from SSO**: (Optional) Enable to automatically create users who don't exist
- **Default user role**: (Required if auto-create enabled) Select role for new users
## Google Workspace
Google Workspace (Google Cloud) configuration for listmonk SSO integration.
### 1. Create a new OIDC provider in Google Cloud Console / Google Workspace
In the Google Cloud Console interface, create a new Project.
- **Project Settings**:
- **Project name**: `Listmonk` (or any preferred name)
- **Branding Settings**:
- **App name**: `Listmonk` (or any preferred name, this will be visible to the users.)
- **Authorised domains**: `listmonk.example.com` (or domains that your instance is available on.)
After creating the project, goto **Clients**.
### 2. Create an client in project.
Create a new client and configure it.
- **Application Settings**:
- **Application type**: `Web application`
- **Name**: `listmonk` (or any preferred name)
- **Authorised JavaScript origins**: `https://listmonk.example.com` (or domains that your instance is available on.)
- **Authorised redirect URIs**: `https://listmonk.example.com/auth/oidc` (or domains that your instance is available on, value is also available in the Settings mentioned above. (Redirect URL for oAuth provider))
Hit save and note the Client ID and Client Secret
### 3. Configure listmonk
In listmonk Admin → Settings → Security → OIDC:
- **Enable OIDC SSO**: Turn on
- **Provider URL**: `https://accounts.google.com` (select Google to Auto-Fill)
- **Provider Name**: Set a name to show on the login form (e.g., `Login with OrgName`)
- **Client ID**: Client ID copied from Console (e.g., `XXXX.apps.googleusercontent.com`)
- **Client Secret**: Client Secret copied from Console
- **Auto-create users from SSO**: (Optional) Enable to automatically create users who don't exist
- **Default user role**: (Required if auto-create enabled) Select role for new users
================================================
FILE: docs/docs/content/querying-and-segmentation.md
================================================
# Querying and segmenting subscribers
listmonk allows the writing of partial Postgres SQL expressions to query, filter, and segment subscribers.
## Database fields
These are the fields in the subscriber database that can be queried.
| Field | Description |
| ------------------------ | --------------------------------------------------------------------------------------------------- |
| `subscribers.uuid` | The randomly generated unique ID of the subscriber |
| `subscribers.email` | E-mail ID of the subscriber |
| `subscribers.name` | Name of the subscriber |
| `subscribers.status` | Status of the subscriber (`enabled`, `disabled`, `blocklisted`) |
| `subscribers.attribs` | Map of arbitrary attributes represented as JSON. Accessed via the `->` and `->>` Postgres operator. |
| `subscribers.created_at` | Timestamp when the subscriber was first added |
| `subscribers.updated_at` | Timestamp when the subscriber was modified |
## Sample attributes
Here's a sample JSON map of attributes assigned to an imaginary subscriber.
```json
{
"city": "Bengaluru",
"likes_tea": true,
"spoken_languages": ["English", "Malayalam"],
"projects": 3,
"stack": {
"frameworks": ["echo", "go"],
"languages": ["go", "python"],
"preferred_language": "go"
}
}
```
## Sample SQL query expressions
#### Find a subscriber by e-mail
```sql
-- Exact match
subscribers.email = 'some@domain.com'
-- Partial match to find e-mails that end in @domain.com.
subscribers.email LIKE '%@domain.com'
```
#### Find a subscriber by name
```sql
-- Find all subscribers whose name start with John.
subscribers.email LIKE 'John%'
```
#### Multiple conditions
```sql
-- Find all Johns who have been blocklisted.
subscribers.email LIKE 'John%' AND subscribers.status = 'blocklisted'
```
#### Querying subscribers who viewed the campaign email
```sql
-- Find all subscribers who viewed the campaign email.
EXISTS(SELECT 1 FROM campaign_views WHERE campaign_views.subscriber_id=subscribers.id AND campaign_views.campaign_id=**WARNING:**This permission allows execution of arbitrary SQL expressions and SQL functions. While it is readonly on the table data, it allows querying of all lists and subscribers directly from the database superceding individual list and subscriber permissions. Raw SQL expressions make it possible to obtain Postgres database configuration and potentially interact with other Postgres system features. Give this permission ONLY to trusted users. [Learn more](#subscriberssql_query). | | | tx:send | Send transactional messages to subscribers | | campaigns | campaigns:get | Get and view campaigns belonging to permitted lists | | | campaigns:get_all | Get and view campaigns across all lists | | | campaigns:get_analytics | Access campaign performance metrics | | | campaigns:manage | Create, update, and delete campaigns | | bounces | bounces:get | Get email bounce records | | | bounces:manage | Process and handle bounced emails | | | webhooks:post_bounce | Receive bounce notifications via webhook | | media | media:get | Get uploaded media files | | | media:manage | Upload, update, and delete media | | templates | templates:get | Get email templates | | | templates:manage | Create, update, and delete templates | | users | users:get | Get system user accounts | | | users:manage | Create, update, and delete user accounts **WARNING:**This permission allows creation of users with any role, including Super Admin. This permission should only be given to Super Admin level accounts | | | roles:get | Get user roles and permissions | | | roles:manage | Create and modify user roles | | settings | settings:get | Get system settings | | | settings:manage | Modify system configuration | | | settings:maintain | Perform system maintenance tasks | ## List roles A list role is a collection of permissions assigned per list. Each list can be assigned a view (read) or manage (update) permission. List roles are attached to user accounts. Only the lists defined in a list role is accessible by the user, be it on the admin UI or via API calls. Do note that the `lists:get_all` and `lists:manage_all` permissions in user roles override all per-list permissions. ## API users A user account can be of two types, a regular user or an API user. API users are meant for intertacting with the listmonk APIs programmatically. Unlike regular user accounts that have custom passwords or OIDC for authentication, API users get an automatically generated secret token. ## `subscribers:sql_query` This permission allowers users to write and execute arbitrary SQL queries on the database. Although it is executed as a read-only transaction disallowing changing of data in the database tables, it allows querying of all lists, subscribers and other data directly from the database superceding individual list and subscriber permissions. Raw SQL expressions also make it possible to obtain Postgres database configuration and potentially interact with other Postgres system features. Give this permission ONLY to trusted users. If this permission is being assigned to many users, it is highly recommended that you create a custom Postgres role disallowing any privileged operations. For example: ```sql CREATE ROLE listmonk_app WITH LOGIN PASSWORD '...' NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION; ``` ================================================ FILE: docs/docs/content/security-reports.md ================================================ If you spot a security vulnerability in listmonk, please report it via GitHub [security advisories](https://github.com/knadh/listmonk/security/advisories). ### What not to report The below listed scenarios are either not security vulnerabilities or are of acceptable risk. They keep getting reported unfortunately. Please refrain from doing so. ### SQL injection via subscriber query The subscribers UI (and APIs) support issuing of arbitrary SQL expressions via a `query` parameter. While listmonk ensures that the queries are executed as readonly and has basic checks for target tables to prevent accidental side-effects, it is not really possible to prevent arbitrary Turing-complete SQL expressions from calling various Postgres functions. Postgres itself does not offer an easy way to allow/disallow specific functions. That's why this feature is behind a special permission `subscribers:sql_query` and its risks are [clearly documented](__https://listmonk.app/docs/roles-and-permissions/#user-roles__). In a multi-user scenario, it is up to an admin to allow this permission to trusted users. ### Stored XSS via SVG In addition to images, listmonk allows uploading of arbitrary file types, .html, .js, .svg, .* and does not transform or modify the files. That means, it is possible to have `