Back to the show

AI & Tech Daily

AI Takes On Longer Tasks, While Judgement Remains the Hard Part

18:45

METR's latest evaluation shows rapid gains in AI task capability and a revealing gap when reliable feedback disappears. OpenRouter removes providers following data investigations; Stripe opens controlled money-moving tools for agents; Tencent releases Hy4 weights; and MedChemBench separates reading chemistry from choosing compounds. Also: proposed US disclosure rules for AI financing, a quantum experiment in location verification, and GPU-powered event functions on AWS Lambda.

Full transcript

Read the episode.

I'm Jesse Owen. This is AI and Tech Daily.

Longer Tasks, Harder Judgements

An AI agent can be strong at completing a coding task and much weaker at deciding which problem deserves solving. New results from METR put that gap alongside a striking pace of improvement.

Our main story today is METR's latest capability evaluation, and where organisations can usefully hand over more substantial work. The central question is how much the task depends on getting a clear signal that the answer is right.

On 20 November, METR published evaluations of public models with harder tasks designed to separate coding and optimisation from work requiring judgement without quick feedback. Across its two combined task suites, the estimated task horizon doubled every 3.61 months. That's a rapid increase in the difficulty of work these systems can handle.

The measurement needs one brief explanation. A fifty per cent time horizon describes a task's difficulty in estimated human working time, at the point where an agent succeeds half the time. If a task would take a person several hours, that human effort gives the task its place on the scale. The measure describes task difficulty, separately from an agent's unattended runtime.

That distinction changes how to read the progress. Models are succeeding on more demanding assignments. You still need to know the success rate and the kind of assignment before translating that into a business workflow. Half the attempts succeeding may be useful for generating candidate solutions that are cheap to check. It leaves a different amount of work for the reviewer when each answer requires lengthy investigation.

METR found that models usually matched or exceeded the recruited skilled humans on coding and optimisation tasks. Those are substantial capabilities. Where the result can be measured, an agent has something concrete to work towards. A test can expose a broken implementation. An optimisation target can tell you whether a change improved the result.

The harder part appeared when reliable feedback was absent. There, models looked weaker than experts. Choosing a research direction, for example, asks you to judge which question is promising before there's an answer to inspect. The destination itself is part of the work.

For an organisation, I'd use that distinction to separate two decisions that often get bundled together: who sets the objective, and who carries out the implementation. You might delegate a substantial implementation while keeping the choice of objective with the person who understands the business or research problem. That arrangement can still remove a great deal of work.

It also gives a more useful way to evaluate an agent internally. A demonstration with a clear target tells you something about execution. To assess judgement, you need work where choosing the target is genuinely part of the assignment. Otherwise, you can end up measuring the easier half of the job and applying the result to the whole role.

The pace estimate has a wide statistical range: roughly two to five months for a doubling. Only nine per cent of the human-time estimates used direct human baselines, and METR notes that benchmark settings can overstate capability in everyday work. That makes the trend more useful than treating the precise doubling interval as a staffing timetable.

My takeaway is that verifiable implementation is becoming a stronger candidate for delegation. The quality of the feedback available to an agent may tell you more about its usefulness on your next project than its overall position on a leaderboard.

OpenRouter Changes Its Provider Routes

That leaves plenty of room for useful automation. The next development concerns where the data behind that work travels.

On 22 November, OpenRouter said it had removed three inference providers and withdrawn Zero Data Retention eligibility from 125 endpoints after investigating compromised data and failures to uphold data policies. Those endpoint withdrawals include precautionary measures. OpenRouter says its own platform wasn't breached.

For customers, the immediate change is in the available routes and privacy options. An endpoint that previously qualified for a no-retention route may no longer qualify, so the set of services available under that setting has changed. OpenRouter says it has notified potentially affected users and is adding ongoing checks of providers, including checks for undisclosed forwarding.

The investigation used an unusually tangible method. Between July and November, security company Dragos seeded roughly two million prompts with traceable web addresses and credentials. These acted as markers that could reveal activity involving the submitted material. Dragos reported evidence of compromise at three providers, alongside additional failures in data handling.

There's also a specific provider disclosure. On 21 November, Together AI acknowledged theft of stored prompt samples from June and July. It said an AI coding agent had exposed a repository credential. That links a development-system credential to the exposure of historical inference data: the prompts had already been collected and stored when the theft occurred. Together's review of the exposed contents is continuing.

OpenRouter hasn't named the three providers it removed, so its notice doesn't give customers a complete public list of those incidents. The useful customer information is the notification about their own potential exposure and the changed eligibility of routes they use.

For organisations buying inference through an intermediary, I'd treat the full route as part of the service being purchased. The company providing the interface and the company processing a prompt can have different roles. Any further forwarding adds another participant whose handling of that data matters.

The ongoing checks are therefore a meaningful part of OpenRouter's response. They can help keep routing decisions tied to how providers handle requests in practice, including after an endpoint first becomes available. For a business sending sensitive material, that continuing visibility is valuable.

Stripe Gives Agents Transaction Authority

There's a more deliberate kind of access opening up in business finance, with approval built into the transaction.

Stripe has opened public-preview write capabilities for Treasury agents. Announced on 19 November, the tools let an agent transfer money, convert currencies and pay vendors through Stripe's Model Context Protocol connection. MCP is the interface that lets an AI system call tools exposed by another service.

The practical change is that financial analysis can lead into execution within Stripe. An agent's recommendation no longer has to end with someone taking the details into a separate payment workflow. For an eligible business, that could reduce the handoff between deciding what to pay and preparing the action.

Stripe offers two access arrangements, and their differences are important. An agent connected through a user's OAuth authorisation inherits that user's permissions. Every money-moving action requires confirmation. The agent can therefore work within an existing user's access while leaving an explicit approval step at the point money moves.

Dedicated agent API keys provide persistent access and support custom approval rules. Refunds, payouts and outbound transfers require human approval by default. That gives businesses a separate way to define an agent's continuing access, with rules suited to the work it's allowed to carry out.

You can see why these arrangements suit different uses. A person working interactively with an assistant can confirm a proposed action as part of the conversation. An ongoing business process needs a durable identity and a defined approval arrangement that persists beyond that conversation. Stripe is making both patterns available.

For the person responsible for finance, the useful design question is which transactions the agent can prepare, and which approvals apply when it tries to execute them. The available access methods make those choices explicit. They also let an organisation separate permission to analyse information from permission to move funds.

Availability follows the underlying Treasury capabilities and the market the business operates in. This is a public preview for eligible accounts, so access needs to be assessed against that actual setup.

I see the strongest opportunity in removing repetitive payment preparation while keeping transaction authority legible to the business. An agent that can carry a decision through to an approved action has a more substantial operational role than one whose work ends with a recommendation.

Tencent Releases Hy4 Weights

For developers who want more control over where a model runs, Tencent has made a substantial release.

On 20 November, Tencent published downloadable weights for Hy4, its flagship multimodal model. The release includes instruction-tuned weights, an FP8 version and a pretrained base. That gives organisations material they can deploy and modify themselves, alongside the engineering work needed to operate it.

Tencent specifies a language backbone with 770 billion parameters, with 49 billion active for each text token. Parameters are the learned numerical values in the model. The active count describes the portion used for an individual token; the overall model remains a large deployment to accommodate.

Hy4 supports image and video inputs, and its context window is just over one million tokens. That's the amount of material the model can hold in a single working context, measured in the pieces of text it processes. For developers working with long inputs and visual material, the combination opens up a broad range of possible applications on infrastructure they control.

The licence deserves a concrete look before anyone designs a service around it. Hy4 uses Tencent Hy License 1.0. Model-as-a-service operators need separate written permission if worldwide revenue exceeds fifty million US dollars in any consecutive twelve months. They also need permission before using the model in a product or service with more than one hundred million monthly active users.

Strictly internal use has an exemption from those additional commercial terms, provided the model, derivatives and outputs aren't made available to third parties. The inclusion of outputs makes the intended use especially relevant: keeping the weights inside your organisation is only one part of that condition.

For a developer evaluating Hy4, I'd put deployment control and the service's intended audience near the top of the assessment. You can choose where the model runs and modify it, while hardware requirements and licence terms shape what is practical to offer.

The release gives organisations another route to operating a large multimodal architecture themselves. Its value for a particular application will come from how well that application runs on the infrastructure the organisation can supply. With a model this large, the ability to download the weights is the beginning of the deployment decision.

Reading Chemistry and Choosing Compounds

A new chemistry benchmark offers a useful way to see where model abilities separate within one scientific job.

OpenAI introduced MedChemBench on 20 November. It contains 99 problems drawn from 117 papers and patents, testing structure reconstruction, scientific reasoning, compound selection and numerical property prediction. The aim is to examine how models connect molecular structures and experimental evidence to medicinal-chemistry decisions.

OpenAI reports that GPT-6 Astra, at its xhigh reasoning setting, earned 95.4 per cent of the available credit for reconstructing structures. For compound selection and numerical property prediction, it earned about half the available credit. Those component results give a much more informative picture than a single overall score.

Reconstructing a structure is a task of accurately recovering the chemical information in front of you. Choosing a compound asks a different question: given the evidence, which candidate looks promising enough to pursue? Predicting a numerical property adds the demand for a specific estimate. A model can be very good at the first step and still struggle with those later decisions.

That's a useful distinction for a research organisation considering where assistance could fit. Accurate extraction and interpretation of structures could help with the work of handling published material. A system used to prioritise synthesis would need strength in the selection task itself. The score for reading the input can't answer that second question.

The evaluation used historical published material, with no tools, feedback or new experiments available to the models. Prior training exposure can't be ruled out. Those conditions make this a view of model performance on existing scientific evidence; laboratory decision-making involves information and feedback that this test didn't supply.

I find the breakdown more useful than a broad claim that AI can do chemistry. It gives researchers a way to locate the difficult step. If a model has recovered the structures accurately but chooses poorly between compounds, improving the reading stage alone won't solve the selection problem.

For organisations working on drug research, the promising opportunity is assistance with specific parts of the scientific workflow. The next demanding capability is using that understanding to make better choices about what to investigate experimentally. MedChemBench gives that challenge a more precise shape.

A Proposal to Expose AI Financing

A shorter development from Washington puts the financing behind AI expansion under a more specific spotlight.

Four US senators introduced the bipartisan AUDIT AI Act on 19 November. The proposal would require additional disclosures from public companies about AI governance, infrastructure financing and material AI risks through Securities and Exchange Commission rules.

The financing provisions include joint ventures and special-purpose vehicles. Those are arrangements that can place projects and their obligations in separate entities. Disclosing them would help investors understand how infrastructure commitments are organised beyond the headline total attached to an investment announcement.

The bill also addresses dependence on third-party models, concentration of compute, costs and data liabilities. These are practical questions about what a company's AI plans rely on and where obligations sit.

This is proposed legislation, so it creates no new compliance requirement today. Passage, final wording and implementation timing remain unresolved.

For executives, the signal is that lawmakers want a clearer account of who carries the obligations behind AI spending. My reading is that the quality of those explanations could become increasingly important to investors assessing an expansion: how much has been committed, through which entity, and what the company depends on to make the investment useful.

Quantum Proof of Location

And here's a security experiment with a very physical question: is a device where it says it is?

NIST researchers and collaborators have demonstrated quantum position verification in a laboratory, using entangled photons, response timing and a loophole-free Bell test. The work was published on 18 November, with verifying stations 195 metres apart.

Entangled photons have linked measurement outcomes. The experiment combines those quantum correlations with the timing of responses to test a device's claimed location. A Bell test checks for correlations that go beyond what a classical explanation permits. The location result comes from the protocol as a whole, bringing quantum measurements and timing together.

The security result has a specific boundary: the demonstrated attacker model allows only weak entanglement between attackers before each trial. That's an important condition for researchers assessing what the experiment establishes.

NIST puts practical applications potentially more than a decade away, with high-fidelity quantum communication networks needed to support them. So the immediate value is an experimental foundation that researchers can build on.

The intriguing possibility is adding physical origin to the information used to authorise access. A sensitive system could eventually ask for evidence that a request comes from an allowed location as well as checking identity. I'd see that as a potential extra layer for systems where place genuinely matters, with considerable network engineering still between this laboratory demonstration and a deployable service.

What Changes for You

Back at the developer's desk, there's a new way to connect an uploaded file to GPU processing on AWS.

AWS added GPU-accelerated Event Functions on Lambda Managed Instances on 18 November. Developers can use them for inference, image processing and other GPU work inside event-driven applications. An incoming event can trigger the compute without the developer operating a separate GPU-serving stack.

The underlying machines are EC2 G-series instances in the customer's account. This is a capability of Lambda Managed Instances, separate from ordinary Lambda execution. AWS manages drivers, instance lifecycle and scaling, and supplies container images prepared for supported frameworks.

Each execution environment receives one GPU or a fraction of a GPU. Accelerator-memory settings run from three to 48 gigabytes. The memory range and single-GPU allocation limit the size of a job that can run in one environment.

Access is available in commercial regions where both Lambda Managed Instances and the relevant G-series instances exist. Builders still need container packaging and cloud engineering skills, but AWS takes on more of the machinery underneath the event function.

The bill is based on EC2 instance usage, plus a 9.75 per cent management fee and twenty US cents per million requests. Eligible EC2 discounts apply, and scaling to zero is available. The cost therefore follows allocated instances, which is a different calculation from paying only for milliseconds of ordinary function execution.

For an AWS builder with event-driven inference work, I'd see this as a useful option when maintaining the GPU-serving infrastructure is taking too much effort. The economic question is utilisation: how well the incoming jobs use the instances being paid for. Scaling to zero can help with quiet periods, while memory needs and the pattern of arriving work shape the configuration. The convenience has a clear price, giving developers something concrete to compare with the cost of running that infrastructure themselves.

You'll find the sources and full transcript at owenonthenet.com. Thanks for listening.

Sources

Reporting behind this episode.

  1. metr.org/blog/2026-11-20-q4-2026-capability-update
  2. openrouter.ai/blog/announcements/data-processing-incidents-on-inference-providers
  3. dragos.com/blog/ai-supply-chain-security-canary-tokens
  4. together.ai/blog/security-event-prompt-leakage-for-requests-in-june-and-july-of-2026
  5. stripe.com/blog/give-your-agents-controlled-access-to-analyze-your-data-and-move-funds
  6. docs.stripe.com/treasury/agents
  7. huggingface.co/tencent/Hy4
  8. huggingface.co/tencent/Hy4/raw/main/LICENSE
  9. huggingface.co/api/models/tencent/Hy4/commits/main
  10. openai.com/index/introducing-medchembench
  11. warner.senate.gov/newsroom/press-releases/warner-kennedy-alsobrooks-rounds-introduce-bipartisan-legislation-to-increase-transparency-and-oversight-of-ai-investments
  12. nist.gov/news-events/news/2026/11/quantum-physics-can-verify-persons-location-nist-researchers-show
  13. nature.com/articles/s41586-026-11247-3
  14. aws.amazon.com/about-aws/whats-new/2026/11/aws-lambda-gpu-managed-instances
  15. docs.aws.amazon.com/lambda/latest/dg/lambda-managed-instances-gpu.html