On October 5, I wanted a support mailbox to receive copies of the transactional emails we sent to customers. The application change eventually came down to adding a static BCC address at selected send points.
The work around that line was much larger: finding the real senders, preserving exclusions, checking the installed email library, and understanding how each repository reached production. One deployment workflow also contained an artifact download with no corresponding upload.
This is an engineering note about that investigation and the release discipline it calls for. The available record establishes prepared application changes and a diagnosed workflow problem. It does not establish a verified production rollout.
Find the senders before editing them
The first obvious place to look was the dedicated email service. It covered welcome messages and order confirmations. That was only part of the picture.
I asked for another repository to be checked because it handled operational workflows. It also contained customer-facing email, including manually triggered correspondence, in a directory whose name suggested backup behavior. A separate service emitted events that could eventually lead to email without calling the email provider itself.
Repository names were an unreliable map of the system.
For a change like this, build a short inventory before editing:
- Which functions construct an email payload?
- Which wrappers or event handlers reach those functions?
- Which deployed service runs each path?
- Which messages need the additional recipient?
- Which messages must remain unchanged?
Dependency searches help locate candidates. Call-site inspection establishes what they actually do. Event producers need tracing to their consumers; an upstream service may require no modification even when its work causes an email downstream.
The exclusions mattered as much as the additions. I explicitly excluded automated follow-up and refund-reminder messages. Internal operational alerts also stayed outside the customer-correspondence scope.
A blanket change to every SendGrid call would have been easier to describe and wrong for the requirement. The final review needed a list of intentionally unchanged paths alongside the edited ones.
Check the library that will actually run
The next question was whether the proposed payload matched the installed SendGrid library.
The raw API and a language SDK can expose different input shapes. SendGrid's API documentation represents BCC recipients as email objects inside an array. Application code may use a helper that accepts a simpler representation and serializes it. Confusing those layers can turn a compatible one-line addition into an unnecessary rewrite. SendGrid's personalization documentation describes the API structure.
I asked for the versions in the affected repositories to be checked. Both lockfiles resolved the mail package and its helpers to 7.7.0. That mattered more than the broader version range in the package manifests.
The proposed change kept those dependencies and the existing recipient style intact. Compatibility had to be evaluated against that version, rather than assumed from the latest documentation alone.
A repeatable check is straightforward:
1. Read the manifest and lockfile. 2. Confirm how CI installs dependencies and whether it honors the lockfile. 3. Inspect the resolved helper's handling of the new field. 4. Exercise serialization with that installed dependency. 5. Verify the outgoing payload, including fields that should remain unchanged.
This is a narrow change. Upgrading the SDK or reorganizing the mail layer would introduce separate questions into the same review.
Keep the implementation proportional
The early suggestions included configuration and conditional BCC logic. One version returned undefined when the primary recipient matched the support address. Another omitted the field through a conditional spread.
I pushed back on the extra machinery. The requirement was a fixed mailbox, and the selected implementation became a static BCC address.
An anonymized illustration of the application-side shape is:
1personalizations: [{2 to: customerEmail,3 bcc: 'support@example.com',4 dynamicTemplateData: templateData5}]This is illustrative SDK input, not a complete raw API request. It assumes the application's pinned helper supports that shape and that the support address is not already a recipient.
That second assumption needs attention. SendGrid rejects duplicate recipients across the To, CC, and BCC fields of a personalization. Its validation guidance documents the constraint.
Choosing a static address does not make the duplicate case disappear. Verify whether account rules and the selected call sites exclude it. If the address can occur, define the behavior explicitly: for example, omit the additional BCC when the mailbox already receives the message. Avoid building a general configuration system to answer a single recipient question.
Keep the implementation small and its assumptions visible. Spend review effort proving those assumptions.
Test the boundary and the exclusions
A test that checks whether send() was called is too weak for a recipient change. It can pass while the wrong mailbox receives the message or a previously excluded path acquires BCC.
A useful test set would cover:
- An included transactional message gains the intended BCC.
- Its original recipient, template data, sender, and reply behavior remain unchanged.
- Excluded reminders and internal messages do not gain the address.
- Serialization produces the expected provider payload.
- A duplicate-recipient fixture exercises the chosen policy.
These are recommended checks, not tests claimed to have run during this investigation.
Capture payloads at the provider boundary, using the repository's installed dependency where serialization is relevant. Use synthetic addresses and template data. Tests should not send real customer correspondence merely to confirm an object property.
Also check existing error handling. If the provider rejects the new payload, does the caller report failure accurately? Could an existing retry replay the customer message? A BCC edit should not quietly change the meaning of success or introduce a second independent send.
Read deployment workflows before merging
The affected repositories had different release semantics.
In the email service, merging triggered a development deployment; production required a separate manual workflow dispatch. In the other repository, merging triggered the production deployment workflow directly.
The same Git operation therefore had different operational consequences.
Before merging, record the target branch, trigger, environment, build source, deployment destination, and rollback route for each repository. Read job conditions and dependencies, not just the workflow filename. A file named deploy.yml says very little about which version reaches which environment.
For a multi-repository change, release order also matters. It may be acceptable for some transactional messages to receive BCC before others, but that should be an understood intermediate state. Mark each service separately rather than treating several commits as one indivisible release.
This is where a tiny patch starts requiring real operational attention. The application diff cannot tell you whether clicking Merge updates a development function or customer-facing production code.
Follow the artifact from producer to consumer
The workflow investigation exposed two separate problems.
First, an old artifact-deletion step failed before the build, causing later steps to be skipped. Comparing neighboring repositories showed that some had already removed or disabled that pattern.
Second, the production job attempted to download a GitHub Actions artifact that the development job never uploaded. The package was being uploaded to object storage instead.
Those are different storage paths. A ZIP in object storage does not satisfy a GitHub Actions artifact download. GitHub's artifact documentation describes the upload-and-download relationship used to transfer workflow outputs.
Removing the first failing step would therefore expose another likely failure rather than establish a working release.
The useful question was: where is this deployment's package produced, and how does the deployment job obtain that exact package?
For every consumer, identify its producer, artifact name or key, source revision, and expected lifetime. If a workflow rebuilds during promotion, it should be clear that this is a new build. If it promotes an existing package, identify the precise version rather than relying on an ambiguous latest ZIP.
The investigation proposed removing stale steps to align the workflow with its actual object-storage path. At the point documented, that diagnosis had not yet been applied. A proposed fix belongs in the release record as pending work.
Require evidence at each stage
A practical rollout record should distinguish four outcomes:
1. Build: the intended revision produced a package and passed the relevant checks. 2. Deployment: the intended environment accepted that package or version. 3. Runtime: the live service reports or otherwise demonstrates the expected revision. 4. Behavior: an authorized controlled message reaches the intended customer test mailbox and BCC mailbox, with exclusions preserved.
Each outcome needs its own evidence. A green build cannot establish mailbox receipt. A successful deployment job cannot establish that every relevant sender was included.
Before rollout, identify the last known-good package or revision and how to restore it. Define rollback triggers such as recipient validation failures or unexpected copying. If anything is copied incorrectly, rolling back stops future behavior; it cannot recall information already delivered.
Keep verification data minimal. Release revision, message category, provider request identifier, and delivery outcome are often more useful than logging full email bodies. Limit access and retention for whatever evidence is collected.
BCC can give support useful correspondence context. It also creates another copy of customer content. Mailbox access, retention, deletion handling, and whether that copying is appropriate remain separate responsibilities. BCC alone does not provide a complete audit trail or resolve privacy requirements.
Define done beyond the diff
The lasting lesson from this work is concrete: estimate a small change by the behavior it must establish across the system.
Here, that meant a verified sender inventory, explicit exclusions, compatible payloads, repository-specific release plans, a coherent artifact path, and evidence from the running service. The code could remain tiny while the completion criteria stayed demanding.
When those criteria are missing, the honest status is precise: patch prepared, workflow diagnosed, deployment pending, or runtime verification outstanding. That precision is part of delivering the change.