Portfolio optimisation in a changing market

teaching

Motivation

A forecast can be statistically accurate and still lead to an unprofitable trading strategy. The predicted gain may be too small to cover trading costs, several attractive investments may share the same risk, or the forecasting relationship may have changed by the time it is estimated. An optimiser can also amplify estimation errors by placing large positions in directions that appear unusually profitable or safe.

These interactions make portfolio management a problem in both optimisation and inference. A long history supplies information, but recent observations may be much more relevant to the next decision. Changing a portfolio can reduce risk or exploit a new opportunity, while also consuming part of the expected profit (trading costs).

Project goal

Develop a simulated market and investigate how to learn and trade within it. The model below is a suggested starting point: it combines changing predictive relationships, correlated returns, changing volatility, and trading costs with relatively little mathematical machinery. You may adapt it to the question you choose, explaining what each modification changes. The forecasting method, portfolio objective, optimisation approach, and experimental design are open choices.

A suggested simulated market

Treat time as a sequence of identical trading days, with 365 days per simulated year. This convention removes calendar details while retaining the distinction between fast and slow changes. For example, you could generate 50 years of historical observations and then trade for two further years, continuing to learn as new observations arrive. The market states continue across this boundary; the trading account begins with its chosen initial capital.

Returns, predictors, and shared risk

Consider \(N\) stocks with \(p\) observed characteristics per stock (eg. price, volume, momentum, etc…) and \(K\) common risk factors (eg. market, sector, style, etc…). Values such as \(N=50\), \(p=10\), and \(K=3\) give one possible scale, rather than required dimensions. Let \(X_t\in\mathbb{R}^{N\times p}\) contain the current characteristics, \(\theta_t\in\mathbb{R}^p\) their hidden forecasting coefficients, and \(B\in\mathbb{R}^{N\times K}\) fixed factor loadings. One possible law for the vector of daily arithmetic returns \(r_{t+1}\in\mathbb{R}^N\) is

\[ r_{t+1}=X_t\theta_t+s_t\bigl(Bf_{t+1}+\sigma\varepsilon_{t+1}\bigr), \qquad f_{t+1}\sim\mathcal{N}(0,I_K), \quad \varepsilon_{t+1}\sim\mathcal{N}(0,I_N), \]

where \(s_t>0\) multiplies volatility and \(\sigma>0\) sets the stock-specific noise scale. A return of \(0.01\) means a one-percent gain over one step. The columns of \(B\) describe shared responses to unexpected shocks; their magnitudes include the factor volatility scales. A column with positive loadings across stocks can represent market-wide risk, while other columns can distinguish groups of stocks. Choose or draw \(B\) once per simulated market.

Predictors and risk factors have different roles. The observed characteristics help forecast returns, while the factor shocks produce unexpected movements shared by several stocks. Diversification can reduce stock-specific noise without eliminating these common movements.

Persistent characteristics and changing coefficients

The dynamics can reuse one scalar process:

\[ u_{t+1}=\rho u_t+\sqrt{1-\rho^2}\,z_{t+1}, \qquad z_{t+1}\sim\mathcal{N}(0,1), \quad u_0\sim\mathcal{N}(0,1). \]

For \(0<\rho<1\), this process has stationary mean zero and variance one. Writing \(\rho=2^{-1/h}\) makes \(h>0\) its autocorrelation half-life in days. Changing persistence therefore does not also change the stationary variance.

Each characteristic \(x_{i,j,t}\) can follow an independent copy, with predictor-dependent half-life \(h_j\). Half-lives ranging from a few days to a few months create opportunities with different durations. These are abstract candidate predictors, observed exactly; they need not be given names such as value or momentum, which would require additional economic definitions.

For each predictor, let

\[ \theta_{j,t}=\bar\theta_j+\tau_j a_{j,t}, \qquad \tau_j\geq0, \]

where \(a_{j,t}\) is another copy of the same process, possibly with a half-life of several months or years. The baseline \(\bar\theta_j\) is drawn once, independently of the dynamic states, and retained throughout a market history. The coefficient is shared across stocks, so cross-sectional observations supply information about its current value. A predictor with \(\bar\theta_j=\tau_j=0\) has no direct predictive effect in this independent-characteristic model. Some candidate predictors can be inactive, with their identities hidden from the trader.

The baseline gives old observations continuing value, while the changing component makes recent observations useful for tracking current conditions. A relationship can weaken or reverse sign without losing its long-run baseline. The size of \(\tau_j\) relative to \(|\bar\theta_j|\) determines how substantial those changes are. Neither the predictor half-life nor the coefficient half-life directly determines how long a position should be held once risk and costs are included.

Changing volatility and available information

Let \(v_t\) be another stationary unit-variance process of the same form and set \(s_t=\exp(\alpha v_t)\), with \(\alpha\geq0\). Persistent \(v_t\) creates sustained periods of higher or lower risk; \(\alpha=0\) gives constant volatility. The multiplier has median one, rather than mean square one, so changing \(\alpha\) also changes average return variance unless the scale is adjusted. All initial Gaussian states and innovation sequences are independent in this starting model, including the return shocks.

Conditional on the complete current simulator state, the return moments are

\[ \mu_t=X_t\theta_t, \qquad \Sigma_t=s_t^2\bigl(BB^\top+\sigma^2I_N\bigr). \]

These formulas permit direct checks of the generator and comparisons with strategies given additional information. They are not the predictive moments conditional on the trader’s observations: uncertainty about the coefficients and volatility remains part of that prediction problem.

Before choosing the position for \(r_{t+1}\), the trader observes \(X_t\), earlier characteristics, returns through \(r_t\), trading costs, and its own account. Realised hidden coefficients, volatility states, future shocks, and the random seed used to generate the deployment path are unavailable to the trading procedure. For an estimation experiment, the realised factor loadings can also remain hidden. The model family can be known without revealing these realised quantities; any extra information supplied to a comparison strategy should be explicit.

Trading costs and accounting

A positive position buys exposure to a stock; a negative position represents a short position, which gains when its price falls and loses when it rises. Let \(W_t>0\) be wealth immediately before trading, \(w_t^-\in\mathbb{R}^N\) the current stock-dollar holdings divided by \(W_t\), and \(w\in\mathbb{R}^N\) the chosen target exposures in the same units. For the trade \(\Delta w=w-w_t^-\), a possible cost as a fraction of \(W_t\) is

\[ C(\Delta w)=c\lVert\Delta w\rVert_1 +\frac{\eta}{2}\lVert\Delta w\rVert_2^2, \qquad c,\eta\geq0. \]

The linear term charges for the total amount bought and sold, often called turnover. The quadratic term makes large changes disproportionately expensive. Both can make an immediate move towards a newly preferred portfolio less attractive than a smaller adjustment.

Accounting is separate from the choice of optimiser. Under zero interest on cash and borrowing, target stock holdings are \(W_tw_t\), costs are paid from cash, and

\[ W_{t+1}=W_t\bigl[1+w_t^\top r_{t+1}-C(w_t-w_t^-)\bigr]. \]

The stock holdings carried to the next decision are \(W_tw_{i,t}(1+r_{i,t+1})\); dividing by \(W_{t+1}\) gives the next pre-trade exposures. They generally differ from yesterday’s targets. Positions, cash, costs, and exposure constraints must use this same convention. Admissible trades must leave positive wealth after paying costs. Nonpositive wealth terminates an account rather than allowing it to continue with undefined wealth fractions.

Strategy constraints

A trading strategy needs a clear description of the positions and actions it is allowed to take. The information and accounting rules above apply throughout; the remaining restrictions should reflect the investment setting you choose. They can also become part of the investigation.

Shorting and borrowing. Are negative stock positions allowed? May the cash balance become negative, meaning that the strategy borrows to finance its holdings? If borrowing is prohibited, purchases and costs must be covered by available funds.
Total exposure and concentration. Specify a finite limit on total stock exposure relative to wealth, and consider whether individual positions also need limits. Total exposure counts the absolute sizes of both long and short positions: offsetting positions can have little net investment while still carrying substantial risk. The choice of limits is open, and their effects can be studied.
Trading restrictions. Limits on trading frequency or on the amount traded may matter for a particular question. If introduced, their consequences should be distinguished from those of transaction costs.

State when restrictions are checked and how breaches caused by price movements are handled. A limit based on estimated risk must use information available to the trader. For a comparison of trading methods, common constraints help distinguish better decisions from greater freedom to take risk. If the constraints themselves vary, the comparison concerns that change as well.

Some possible questions

How much history remains useful? Can an estimator learn durable relationships from old data while adapting to current deviations? When does forgetting help, and what information does it discard?
Does better prediction produce better trading? Forecasts differ in persistence, uncertainty, and the turnover they induce. Could a less accurate predictor lead to a better portfolio after costs?
How does optimisation react to estimation error? Which errors cause the largest changes in positions? Could shrinkage, regularisation, or constraints improve decisions, and when do they become too conservative?
When is it worth trading? How do signal persistence, existing positions, and costs affect the timing and size of adjustments? When does planning several trades ahead help?
What changes as the market becomes larger? More stocks provide more observations and diversification opportunities, but also more quantities to estimate. Which effect explains an apparent improvement?
Which model assumptions matter to a conclusion? Correlated predictors, changing factor loadings, nonlinear effects, or occasional permanent coefficient changes could challenge an explanation developed in the initial model.

These are possible directions, with no prescribed sequence or collection of methods. A focused question may emerge from a mathematical example, a paper, or an unexpected simulation result.

Developing your investigation

Parameter choices determine whether the market is learnable, nearly trivial, or too noisy for useful trading. Small individual signals can combine into a strong portfolio, and many stocks can make changing coefficients easier to estimate from each day’s panel. Adding predictors should not inadvertently increase total predictable return simply by adding more independent opportunities. The balance between aggregate signal strength, coefficient movement, shared risk, and costs is therefore part of the modelling investigation. Comparisons using the hidden current state can help diagnose that balance, but do not automatically solve the dynamic trading problem or provide an upper bound on every strategy’s realised performance.

The historical and deployment periods serve different roles. For the illustrative 50-year history and two-year deployment, they contain 18,250 and 730 steps respectively. Choices can be developed using chronological splits of the historical period, while deployment observations are revealed sequentially and may inform subsequent trades. Fresh independent market histories help assess a procedure without repeatedly adapting it to the same evaluation outcomes. Comparing methods on common paths also helps separate method differences from different market experiences.

The computational scale should leave time to follow initial observations with further reading and experiments.

Evaluating a strategy

Choose measures that express what a successful strategy would achieve in your setting. The objective used to construct a strategy is itself a design choice; evaluation should also reveal consequences that this objective may leave out. Possible measures include:

Returns after costs. Growth of wealth over the deployment period, together with the difference between performance before and after trading costs.
Risk and losses. Variability of portfolio returns, large individual losses, and drawdowns, meaning declines from previous wealth peaks. Across repeated histories, the frequency of insolvency may also be informative.
Risk-adjusted performance. A measure such as the Sharpe ratio compares average return in excess of the cash rate with the standard deviation of returns; the cash rate is zero in the suggested model. Such a ratio does not describe every aspect of losses or the wealth path, and can be unstable when estimated from a short period.
Trading activity and exposure. Turnover, total costs, concentration, and the amount of long and short exposure can help explain how a result was obtained.
Prediction and estimation. Forecast errors or discrepancies from hidden simulator quantities can diagnose a strategy’s behaviour. They do not alone establish that its trading decisions are useful.

These are suggestions, not a fixed scorecard. A small set of complementary measures can reveal whether an apparent improvement comes with more risk, more trading, or poorer performance in adverse conditions. State the observation interval and the convention used for any annualised quantity; scaling daily means or variances to a year requires care when returns are serially dependent.

One profitable two-year episode supplies limited evidence. Fresh simulated histories can show the dispersion of outcomes and the uncertainty in differences between methods, including failed accounts rather than only surviving ones. The performance measures and comparison rules should be chosen before inspecting the deployment outcomes used to assess the final procedure.

Simplifications and their consequences

This is a controlled learning and trading model. The 365-day calendar, zero interest, and fixed universe avoid bookkeeping that is peripheral to the proposed questions. Other simplifications have more substantive consequences:

Linear, exactly observed predictors. There is no measurement error or nonlinear predictive mechanism. Methods suited to linear relationships may consequently perform well.
Permanent coefficient baselines. The deviations are stationary fluctuations around durable relationships. Permanent structural changes require a different coefficient law.
Fixed risk directions. A common volatility multiplier changes covariance magnitudes but leaves conditional return-shock correlations unchanged. It does not create crisis-driven correlation changes.
Independent stock characteristics. Predictable returns may have little alignment with shared risk directions, making some opportunities easy to diversify. Shared predictor components or different signal amplitudes can change this property.
Exogenous prices and stylised costs. Trading changes the account but not future market returns. The model omits market feedback, stock-borrow fees, and realistic liquidity constraints; its weight-based cost law also omits the dependence of execution costs on absolute fund size.
Gaussian arithmetic returns. Returns below \(-1\) are possible, violating the limited liability of a stock. This is a daily-return approximation whose tail behaviour needs checking. Such events are model failures to record, not observations to silently redraw or discard. Clipping returns or adopting a positive-price model changes the law and its conditional moments.
No permanent market growth term. The starting model has zero unconditional expected returns. A long-only equity-growth experiment would need an additional mean component.

Which of these limitations matters depends on the question. Changing one assumption can reveal why a conclusion holds, while adding many mechanisms at once can make that explanation harder to establish.