AI Will Be Used Against You
1. Your Exposure, Their Ammunition
A threatening email quotes a password you really used. It claims the sender controls your webcam and demands payment. The password may be genuine. The claimed access may be fiction.
The UK's National Cyber Security Centre describes this intimidation technique in sextortion messages: an old password can come from an earlier breach and make the threat feel credible. Change the password anywhere it is still used and report the message, but do not treat the quoted password as proof that the sender currently controls your device. Read the NCSC guidance.
The examples below show how personal details can become local, believable bait. The named banks, schools, airlines, employers and healthcare institutions are reference points in fictional messages. Their appearance does not assert that they suffered the depicted breach.
Same attack logic, different local bait. Choose a country:
|
You bank with OCBC Illustrative exposure: phone number and banking relationship |
→ | "OCBC Fraud Alert: confirm this S$1,280 transfer now or your account will be temporarily locked." |
|
Your child goes to Raffles Illustrative exposure: family and school context |
→ | "Urgent: your child was involved in an incident at school. Call this number immediately." |
|
You flew SQ to Tokyo last week Illustrative exposure: travel and booking details |
→ | "Singapore Airlines: your flight SQ638 is eligible for a $180 refund. Claim here." |
|
You see Dr Tan at Mt Elizabeth Illustrative exposure: healthcare relationship |
→ | "Your outstanding balance of $420 at Mt Elizabeth is overdue. Pay now to avoid referral." |
|
You earn $14k/month at DBS Illustrative exposure: employer and salary context |
→ | "DBS HR: your March payslip requires verification due to a CPF adjustment. Log in here." |
AI will be used against you
Leaked information plus AI makes personalised attacks easier to prepare and sustain. AI can interpret messy records, connect details across sources, draft plausible messages, translate them and continue a conversation. AI reduces the effort needed to use exposed information. The requirements differ by attack: account takeover needs usable access, and payment fraud needs a payment path. Other harm can arise from disclosure alone, such as exposing private medical information or giving competitors confidential prices.
The message can be false while the details inside it are true. That is why exposure changes the attack.
The six questions
Use the same questions whenever you examine an exposure finding:
Information, access or an observation.
The person or organisation it matches.
The credible attacker action.
The missing conditions.
The plausible consequence.
The response and what remains.
Contents
2. What Attackers Can Obtain
People and organisations deliberately publish information: job roles, product announcements, social posts, investor communications and contact details. Attackers use that context. Cyber exposure begins with information disclosed without the subject's intention or control: breached records, malware logs, leaked documents, credentials, access offers and criminal discussions.
Voluntary and involuntary information combine. A public job title may identify who approves payments. A leaked mobile number gives the attacker a private route to that person. A stolen contract supplies the counterparty, amount and timing.
Five categories of exposed information
Credentials
Passwords, API keys, session material and authentication tokens. Some may still permit access; others remain useful as evidence that the sender possesses real data.
Personal information
Names, email addresses, phone numbers, home addresses, family links, identity numbers, hobbies, routines and biometrics.
Financial information
Payment cards, bank details, invoices, crypto wallets, salary information and transaction context.
Internal material
Emails, contracts, source code, strategy documents, customer records, organisation charts and approval processes.
Infrastructure
Domains, server addresses, cloud configuration, VPN details, exposed services, network maps and system identifiers.
Three different observations
Exposed information
A verified record or document shows that information has circulated outside its intended boundary.
Potentially usable access
A credential, valid session or access offer may enable entry. Its validity, scope and controls still require verification.
Hostile interest
A marketplace post, insider-recruitment request or actor discussion shows attention or intent. It does not by itself prove access or compromise.
A seller claiming to hold company data establishes that the claim was made. A request to recruit an insider establishes hostile interest. An authorised comparison showing that a sample is genuine establishes disclosure of that sample. These findings can demand action, but they do not say the same thing.
3. Why Old Information Remains Useful
Information does not have one expiry date. Its value depends on how an attacker uses it.
Access
A password, token or session may still open something. Validity, account state, scope, expiry, revocation and replay protections determine whether it works now.
Credibility
An old password, card number or address can make a scammer appear to have current access or privileged knowledge.
Targeting context
Past employers, suppliers, hobbies, travel and relationships help select a believable pretext and recipient.
Leverage
Documents, health data, biometrics and private communications can support extortion, embarrassment or commercial pressure long after access is closed.
Four clocks, not one
Captured
When the information left its intended boundary or the credential was stolen.
Circulated
When it was copied, offered, reposted, sold or combined with other material.
Discovered
When the affected organisation or monitoring process first found it.
Still true?
Whether the password, role, relationship, system, address or business condition remains current.
Sounds too academic? A credential discovered today may have been stolen in 2020. The finding is new to you. The theft is not necessarily new. The question is what the material can still do.
Session material needs exact language
A stolen valid session may allow access without repeating the original MFA login. That does not mean every cookie bypasses MFA or grants unrestricted access. Expiry, revocation, application scope, device binding, replay detection, risk policies and requirements to reauthenticate for sensitive actions can interrupt use. Microsoft's token-protection guidance describes these controls.
4. How Exposure Circulates
Exposure does not follow one mandatory pipeline. Material can move directly from a breach to extortion, from an infected device to account access, from a supplier to a private community, or from a paste site into a phishing campaign. A marketplace and AI may be involved. Neither is required.
Copies can move between sources and uses in either direction. Several routes can exist at the same time.
Breaches and dumps
Databases and document collections may be released once, repackaged repeatedly and incorporated into larger compilations.
Forums and private communities
Actors discuss targets, seek partners, recruit insiders, share methods and advertise material. Access and context vary by community.
Marketplaces
Sellers offer credentials, data or access. An advertisement is a source claim until evidence establishes what the seller actually controls.
Paste sites and messaging channels
Fragments can appear quickly, disappear, be mirrored elsewhere and lose the context that would explain their origin.
Finding the same listing in another channel proves circulation. It does not automatically prove a second incident or provide independent corroboration. Sector-wide reporting can tell an organisation what to prepare for; it cannot prove that the organisation itself is affected.
5. What AI Accelerates
AI changes the economics of using exposure. It can reduce the time and skill needed for several parts of an operation:
Interpret
Extract people, relationships, topics and useful details from large or messy collections.
Connect
Relate a leaked address, public role, supplier and recent event into a more complete target picture.
Prepare
Draft messages, identities, documents, audio or video suited to a target and pretext.
Translate
Improve language and adapt wording for another market, while local knowledge and review still affect quality.
Sustain
Help maintain conversations, produce variants and respond faster across a larger set of targets.
The asymmetry matters. An attacker can use machine time to prepare more variants. A defender still has to establish which observation is genuine, what it means and whether action is justified. AI reduces some attacker costs; it does not remove the bottlenecks of access, execution and getting a person or system to act.
Hong Kong deepfake payment fraud, 2024
In the case widely reported in connection with Arup, the Hong Kong Government's official account says the victim received a phishing email inviting them to a confidential group video conference. The conference was prerecorded, created from public video clips and voices, and allowed no interaction. After it ended, the fraudster continued payment instructions through instant messaging. The victim authorised transfers to five local bank accounts and lost about HK$200 million. Read the official account.
The useful lesson is narrower than "video cannot be trusted." A familiar face or voice cannot carry payment authority by itself. The payment request still needs an independent, trusted verification path.
The Contract Redline Phish
An attacker knows the recipient is working with a real supplier or law firm. A message arrives with ordinary language: revised redlines, final NDA comments or a DocuSign request before close of business. The link opens a counterfeit Microsoft 365, SharePoint or DocuSign page and attempts to capture credentials.
Real counterparties, document names, deal timing and job roles make the message fit the recipient's day. If a password is entered, access still depends on whether it is valid and whether MFA, Conditional Access, session controls and detection stop its use.
Virtual kidnapping and crisis calls
The FBI has warned that criminals can use short AI-generated audio clips to impersonate a loved one in a crisis and request immediate financial help or ransom. The pattern combines identity and relationship context with urgency and synthetic audio. The source does not establish how the audio was obtained in every case. Read the FBI/IC3 warning.
A family verification phrase and a call back through a trusted number interrupt the path. They do not remove the voice samples or family details already available.
What one phishing study measured
A study with 101 participants divided recipients across four groups. Arbitrary phishing emails produced a 12% click-through rate; human-expert emails 54%; fully AI-automated emails 54%; and AI emails with a human in the loop 56%. The measured outcome was clicking a link, not credential entry, account compromise or financial loss. The result shows that the tested automated system matched human experts on that measure under those study conditions. It does not establish a universal phishing success rate. Read the paper.
6. When Exposure Becomes a Company Incident
An exposure finding may reveal information that has already been disclosed or a compromise that has already occurred. It may also identify conditions that could enable future harm. A serious finding can justify urgent investigation while its source, scope or current access remains uncertain. Employees, contractors and suppliers connect organisational systems to devices, identities and relationships outside the company's direct control.
Historical credentials, later theft: Snowflake customer accounts
In 2024, Mandiant documented UNC5537 compromising multiple organisations' Snowflake customer instances with customer credentials previously stolen by infostealer malware. Some associated infections dated back to 2020. In the investigated incidents, the affected credentials remained valid, MFA was not enabled on the impacted accounts, and network allow lists were absent. The actor exported customer data, advertised it for sale and attempted extortion.
Mandiant explicitly reported no evidence that the access came from a breach of Snowflake's enterprise environment. The incidents involved compromised customer accounts. It also observed contractor systems used for personal activity among the infection paths. Read Mandiant's account.
This case connects the clocks. Credentials can be stolen years before they are used. It also shows why the company boundary includes contractors and other parties that use company identities. The exposure was not the whole incident. Continued credential validity and missing controls made later access possible.
What turns a signal into an incident decision?
- Relevance: does the material reliably match this organisation, person, supplier, system or relationship?
- Possible severity: what credible consequence could follow if the source claim and required conditions are true?
- Confidence: what do the source, sample, dates and independent evidence actually establish?
A strong company match does not automatically increase severity. A serious but unconfirmed sale claim can justify urgent investigation without being labelled a confirmed compromise.
7. The Uncomfortable Truth and the Response
You cannot reliably recall information that has already been copied and circulated.
You can change a password, revoke a session, remove an exposed service, interrupt a payment, warn an employee and correct a process. Those actions matter. They reduce particular risks and consequences. They do not reach every copy of the original password, contract, address, biometric, message or image.
Individuals can discover some exposure, change reused passwords, enable strong authentication, verify urgent requests through trusted channels, limit unnecessary public information and report fraud. They cannot see or control every breach, malware log, private community, supplier disclosure or copy made by somebody else. Organisations hold more of the relevant systems, evidence and authority, so they carry the larger monitoring and response responsibility.
From Signal to Action
Exposure intelligence must reach a person or process that can do something about it. Collected findings need context, a decision, an authorised response and evidence of what changed.
Specialist exposure-intelligence providers supply collected observations and analysis from breach material, malware logs and other relevant sources. Cyber Intelligence House (CIH), which I founded, is one example. An organisation may use an external provider, its own collection capability or a combination. The coverage and evidence behind each finding must remain explicit.
Signal
Exposure-intelligence services supply findings, source context and observation dates. Internal security monitoring contributes events from the organisation's own systems. A feed, report or alert supplies evidence to investigate; it does not itself establish every claimed consequence.
Interpretation
Analysts or threat-intelligence platforms relate findings to the organisation's identities, assets, suppliers and business context. Security information and event management systems (SIEM) can correlate relevant internal events. Asset inventories and identity directories help establish which systems and people are actually involved. Preserve the distinctions between relevance, severity, confidence and currentness.
Decision and ownership
Incident or case-management tools and IT service-management workflows record the assessment, responsible person, approved action and unresolved questions. Some responses can proceed under an approved playbook; material business decisions and actions outside that authority return to the appropriate human.
Action
Security orchestration, automation and response systems (SOAR), workflow automation or authorised people coordinate action through the systems that control the affected resource. Identity and secrets-management systems can revoke access or rotate credentials; endpoint-security tools can investigate or isolate an affected device; patch, configuration, cloud and network-management tools can change technical exposure. Finance, procurement, legal and management processes handle payment verification, supplier coordination, notifications and policy changes. Choose actions from the evidence and authority, not from a generic checklist.
Verification
Check the actual control or business-process result, record what changed and retain unresolved risks. Closing a ticket or receiving a successful automation response alone does not establish that the risk was reduced. Continued circulation of old material remains distinct from renewed access or a new compromise.
These are functions, not a compulsory shopping list. One provider or platform may perform several of them. A small organisation may use a security provider and a simple case record; a larger organisation may connect several specialist systems. Each handoff still needs a responsible owner.
AI may help interpret genuine findings or coordinate an approved response when it has the required data, tools and authority. Its model knowledge is not a substitute for current exposure collection, and a generated recommendation does not change an account, device or policy by itself.
Can be changed
Credentials, sessions, permissions, exposed services, payment instructions, monitoring rules and business processes.
Remains exposed
Copies of disclosed documents, personal details, biometrics, communications and historical relationships.
Needs continued attention
Unresolved source claims, repeated circulation, impersonation risk, affected counterparties and signs of current access or targeting.
An empty report is not complete visibility
"Nothing found" means nothing was found for the identifiers, sources, time window and collection methods actually covered. It does not prove that no exposure exists. Exposure monitoring observes circulating information and hostile signals. Asset discovery identifies systems and services. Authorised vulnerability testing examines whether those systems can be exploited. They answer different questions.
Reduced risk is not erased exposure. Muting an alert does not remediate the issue underneath it.
8. Turn a Finding into a Decision
Northstar Wholesale and its supplier
Northstar Wholesale is fictional. The assessment below shows how several observations can matter without being forced into one invented incident.
What was observed
What this establishes
- The old employee information still circulates, while the known password condition has been remediated.
- The marketplace post makes a current claim that Northstar contracts are for sale.
- The sample contains genuine confidential business material connected to Northstar and its supplier.
What remains unknown
- Whether the disclosure originated from Northstar, its supplier or another party.
- Whether the seller holds the full advertised collection.
- Whether the seller has current system access.
- Whether the old employee credential has any connection to the contract disclosure.
Credible consequences
A fraudster could impersonate the supplier and request that the scheduled deposit go to a different bank account, quoting the genuine purchase-order reference and payment date. Those details make the request fit an existing transaction. Completing the fraud would still require the changed payment instructions to pass the buyer's controls.
The confidential prices could also give another party a negotiating advantage without Northstar making a payment, clicking a link or responding to a message. The sample establishes disclosure of the information; the observations do not establish that payment fraud or commercial exploitation has occurred.
Proportionate response
- Preserve the post, sample, timestamps and collection context.
- Investigate the disclosure jointly with the supplier and determine where the contract was stored and accessed.
- Review relevant identity, document and transfer activity for evidence of unauthorised access.
- Brief the staff responsible for the supplier relationship and payments on the exposed order details. Require any changed payment instructions to be verified through established supplier contacts and the approved bank-detail change process.
- Retain the verified remediation status of the old password and keep its possible connection to the contract disclosure marked unknown.
Proposed signal-to-action path
- The observed listing and sample supply the signal.
- Authorised comparison with the contract establishes what was disclosed.
- A case owner coordinates investigation with the supplier.
- Security staff review relevant access and document activity; finance applies the verified payment-change process.
- The case records what was checked or changed, what the evidence supports and what remains unresolved.
These are proposed response steps; their results have not yet been verified. The old password remains recorded as remediated unless new evidence changes that assessment.
Revoking access or correcting a process can reduce risk. It cannot remove copies of the contract already held elsewhere. The unresolved source, scope and access questions remain visible until evidence changes them.
Exposure brief
Finding: Person, organisation, supplier or system concerned: Captured / circulated / discovered dates: What is established: Source claims: Plausible attacker use: Conditions required for that use: Possible consequence: Relevance / possible severity / confidence: Unanswered questions: Response owner: Next action (proposed): Action taken: Result verified: Remaining risk: Evidence that would change the assessment: What remains exposed after the response:
Sources and Related Guides
Primary sources used in this draft
- UK National Cyber Security Centre: Sextortion guidance - old passwords used in threatening messages.
- Hong Kong Government: Combating fraud involving deepfake - official account of the prerecorded HK$200 million conference fraud.
- FBI/IC3: Criminals use generative AI to facilitate financial fraud - text, image, audio and video fraud patterns.
- Mandiant: UNC5537 targets Snowflake customer instances - historical infostealer credentials, customer-account compromise, data sale and extortion.
- Heiding et al.: Evaluating automated spear phishing on human subjects - 101-participant click-through study.
- Microsoft: Protecting tokens in Microsoft Entra ID - replay, reauthentication, revocation and token-protection controls.
Further reading on response
- NIST: Incident response recommendations and considerations for cybersecurity risk management.
- Microsoft Sentinel: Automate responses with playbooks - an implementation example, not a product recommendation.
Related guides on this site
- AI Security Primer - identify what an AI system can know, reach and do, and decide which safeguards and evidence it needs.
- Building with AI Agents - turn an idea into independently checked work while the human controls purpose and consequential decisions.
- AI Bots as Managers - establish shared context, repeatable processes and working connections for bot-managed responsibilities under human authority.
Version history
This primer is versioned like software. Published versions receive a dated badge and frozen snapshot.
- 2026-09-05: current. Rebuilt around attacker utility, circulation, evidence interpretation, organisational response and verification. View snapshot →
- 2026-04-08: first edition. Introduced cyber exposure, attacker use of leaked information and the original exposure-monitoring model. View snapshot →
Dr Mikko S. Niemelä - 2026
Last updated: September 5, 2026