Metaculus¶
Metaculus¶
-
class
Metaculus
(api_domain='www', username=None, password=None)[source]¶ The main class for interacting with Metaculus
- Parameters
-
get_question
(id, name=None)[source]¶ Load a question from Metaculus
- Parameters
id (
int
) – Question id (can be read off from URL)name – Name to assign to this question (used in models)
- Return type
MetaculusQuestion
-
get_questions
(question_status='all', player_status='any', cat=None, pages=1, fail_silent=False, load_detail=True)[source]¶ Retrieve multiple questions from Metaculus API.
- Parameters
- Return type
List
[MetaculusQuestion
]
MetaculusQuestion¶
-
class
MetaculusQuestion
(id, metaculus, data, name=None)[source]¶ A forecasting question on Metaculus
- Parameters
- Variables
activity –
anon_prediction_count –
author –
author_name –
can_use_powers –
close_time – when the question closes
comment_count –
created_time – when the question was created
id – question id
is_continuous – is the question continuous or binary?
last_activity_time –
page_url – url for the question page on Metaculus
possibilities –
prediction_histogram – histogram of the current community prediction
prediction_timeseries – predictions on this question over time
publish_time – when the question was published
resolution –
resolve_time – when the question will resolve
status –
title –
type –
url –
votes –
-
static
get_central_quantiles
(df, percent_kept=0.95, side_cut_from='both')[source]¶ Get the values that bound the central (percent_kept) of the sample distribution, i.e., cutting the tails from these values will give you the central. If passed a dataframe with multiple variables, the bounds that encompass all variables will be returned.
- Parameters
- Returns
lower and upper values of the central (percent_kept) of the sample distribution.
-
refresh_question
()[source]¶ Refetch the question data from Metaculus, used when the question data might have changed
-
sample_community
()[source]¶ Get one sample from the distribution of the Metaculus community’s prediction on this question (sample is denormalized/on the the true scale of the question)
ContinuousQuestion¶
-
class
ContinuousQuestion
(id, metaculus, data, name=None)[source]¶ A continuous Metaculus question – a question of the form, what’s your distribution on this event?
-
change_since
(since)[source]¶ Calculate change in community prediction median between the argument and most recent prediction
- Parameters
since (
datetime
) – datetime- Returns
change in median community prediction since datetime
-
community_dist
()[source]¶ Get the community distribution for this question NB: currently missing the part of the distribtion outside the question range
- Return type
PointDensity
- Returns
the (true-scale) community distribution as a histogram.
-
community_dist_in_range
()[source]¶ A distribution for the portion of the current normalized community prediction that’s within the question’s range, i.e. 0…(len(self.prediction_histogram)-1).
- Returns
distribution on integers
-
denormalize_samples
(samples)[source]¶ Map samples from the Metaculus normalized scale to the true scale :param samples: samples on the normalized scale :return: samples from a distribution answering the prediction question
(true scale)
-
property
has_predictions
¶ Are there any predictions for the question yet?
-
property
high_open
¶ Are you allowed to place probability mass above the top of this question’s range?
- Return type
-
property
latest_community_percentiles
¶ - Returns
Some percentiles for the metaculus commununity’s latest rough prediction. prediction_histogram returns a more fine-grained histogram of the community prediction
-
property
low_open
¶ Are you allowed to place probability mass below the bottom of this question’s range?
- Return type
-
normalize_samples
(samples)[source]¶ Map samples from their true scale to the Metaculus normalized scale :param samples: samples from a distribution answering the prediction question
(true scale)
- Returns
samples on the normalized scale
-
property
p_outside
¶ How much probability mass is outside this question’s range?
-
prepare_logistic
(normalized_dist)[source]¶ Transform a single logistic distribution by clipping the parameters and adding scale information as needed for submission to Metaculus. The loc and scale have to be within a certain range for the Metaculus API to accept the prediction.
- Parameters
dist – a (normalized) logistic distribution
- Return type
Logistic
- Returns
a transformed logistic distribution
-
prepare_logistic_mixture
(normalized_dist)[source]¶ Transform a (normalized) logistic mixture distribution as needed for submission to Metaculus.
- Parameters
normalized_dist (
LogisticMixture
) – normalized mixture dist- Return type
LogisticMixture
- Returns
normalized dist clipped and formatted for the API
-
property
question_range
¶ Range of answers specified when the question was created
-
sample_community
()[source]¶ Sample an approximation of the entire current community prediction, on the true scale of the question. The main reason that it’s just an approximation is that we don’t know exactly where probability mass outside of the question range should be, so we place it arbitrarily
- Return type
- Returns
One sample on the true scale
-
sample_normalized_community
()[source]¶ Sample an approximation of the entire current community prediction, on the normalized scale. The main reason that it’s just an approximation is that we don’t know exactly where probability mass outside of the question range should be, so we place it arbitrarily.
- Return type
- Returns
One sample on the normalized scale
-
show_community_prediction
(percent_kept=0.95, side_cut_from='both', num_samples=1000, **kwargs)[source]¶ Plot samples from the community prediction on this question
-
show_prediction
(samples, plot_samples=True, plot_fitted=False, percent_kept=0.95, side_cut_from='both', show_community=False, num_samples=1000, **kwargs)[source]¶ Plot prediction on the true question scale from samples or a submission object. Optionally compare prediction against a sample from the distribution of community predictions
- Parameters
samples – samples from a distribution answering the prediction question (true scale). Can either be a 1-d array corresponding to one model’s predictions, or a pandas DataFrame with each column corresponding to a distinct model’s predictions
plot_samples (
bool
) – boolean indicating whether to plot the raw samplesplot_fitted (
bool
) – boolean indicating whether to compute Logistic Mixture Params from samples and plot the resulting fitted distribution. Note this is currently only supported for 1-d samplespercent_kept (
float
) – percentage of sample distrubtion to keepside_cut_from (
str
) – which side to cut tails from, either ‘both’,’lower’, or ‘upper’show_community (
bool
) – boolean indicating whether comparison to community predictions should be madenum_samples (
int
) – number of samples from the communitykwargs – additional plotting parameters
-
submit_from_samples
(samples, verbose=False)[source]¶ Submit prediction to Metaculus based on samples from a prediction distribution
- Parameters
samples – Samples from a distribution answering the prediction question
- Return type
Response
- Returns
logistic mixture params clipped and formatted to submit to Metaculus
-
LinearQuestion¶
-
class
LinearQuestion
(id, metaculus, data, name=None)[source]¶ A continuous Metaculus question that’s on a linear (as opposed to a log) scale”
-
get_true_scale_logistic
(normalized_dist)[source]¶ Convert a normalized logistic distribution to a logistic on the true scale of the question.
- Parameters
normalized_dist (
Logistic
) – normalized logistic distribution- Return type
Logistic
- Returns
logistic distribution on the true scale of the question
-
get_true_scale_mixture
(normalized_dist)[source]¶ Convert a normalized logistic mixture distribution to a logistic on the true scale of the question.
- Parameters
normalized_dist (
LogisticMixture
) – normalized logistic mixture dist- Return type
LogisticMixture
- Returns
same distribution rescaled to the true scale of the question
-
LinearDateQuestion¶
-
class
LinearDateQuestion
(id, metaculus, data, name=None)[source]¶
BinaryQuestion¶
-
class
BinaryQuestion
(id, metaculus, data, name=None)[source]¶ A binary Metaculus question – how likely is this event to happen, from 0 to 1?
-
change_since
(since)[source]¶ Calculate change in community prediction between the argument and most recent prediction
- Parameters
since (
datetime
) – datetime- Returns
change in community prediction since datetime
-
sample_community
()[source]¶ Sample from the Metaculus community distribution (Bernoulli).
- Return type
-
score_my_predictions
()[source]¶ Score all of my predictions according to the question resolution (or according to the current community prediction if the resolution isn’t available)
- Returns
List of ScoredPredictions with Brier scores
-
score_prediction
(prediction, resolution)[source]¶ Score a prediction relative to a resolution using a Brier Score.
- Parameters
prediction – how likely is the event to happen, from 0 to 1?
resolution (
float
) – how likely is the event to happen, from 0 to 1? (0 if it didn’t, 1 if it did)
- Return type
ScoredPrediction
- Returns
ScoredPrediction with Brier score, see https://en.wikipedia.org/wiki/Brier_score#Definition 0 is best, 1 is worst, 0.25 is chance
-