Skip to content

Data Recombination

Data Recombination (payn.Recombination.DataRecombination)

This class constructs the final balanced datasets required for downstream tasks, for example yield prediction regression. It merges the verified positive examples with the reliable negative examples identified in the previous step, ensuring the correct assignment of target labels.

  • Yield Assignment: Known positives retain their experimentally measured yield values (e.g., 20% to 100%) from the original HTE dataset. Augmented negatives are assigned a synthetic yield value (defaulting to 0.0%) to represent reaction failure.
  • Baseline Comparison: To validate the efficacy of the Spy-based filtering, the module also generates a naïve baseline dataset via generate_unlabeled_as_negative_set. In this baseline, all unlabeled data points are treated as negatives (assigned 0% yield), regardless of their actual likelihood of being latent positives. Comparing the regression performance on the "Augmented" vs. "Naïve" datasets directly quantifies the value added by the PU learning step. Since training on known positives usually outperformed this approach, we selected the “positives-only” as a comparison for our spy PU approach.
  • Integrity Validation: Before returning the recombined dataset, the module triggers validate_split_integrity. This ensures that the concatenation process has not inadvertently dropped rows or introduced duplicates, preserving the total sample count across the workflow.

Recombines datasets by merging spy-infused training data with identified negatives.

This class produces regression datasets by combining known positive examples with negatives. The known positives retain their true yield values (from a specified yield column), whereas the negatives are assigned a fixed negative yield value.

Attributes:

Name Type Description
spy_inf_train_data DataFrame

Spy-infused training data.
augmented_negatives DataFrame

DataFrame containing identified augmented negatives.
known_positives DataFrame

DataFrame containing known positive examples.
random_state int

Random seed for reproducibility.
logger logging

Logger instance for logging artifacts and messages.
known_yield_column str

Column name in known positives containing true yield values.
augmented_target_column str

Column name where the regression target (yield) will be stored.
mod_data_point_role_column str

Column name indicating the role of each data point.
Source code in payn\Recombination\datarecombination.py 
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
class DataRecombination:
    """
    Recombines datasets by merging spy-infused training data with identified negatives.

    This class produces regression datasets by combining known positive examples with negatives.
    The known positives retain their true yield values (from a specified yield column), whereas the negatives
    are assigned a fixed negative yield value.

    Attributes:
        spy_inf_train_data (pd.DataFrame): Spy-infused training data.
        augmented_negatives (pd.DataFrame): DataFrame containing identified augmented negatives.
        known_positives (pd.DataFrame): DataFrame containing known positive examples.
        random_state (int): Random seed for reproducibility.
        logger (Logger, optional): Logger instance for logging artifacts and messages.
        known_yield_column (str, optional): Column name in known positives containing true yield values.
        augmented_target_column (str, optional): Column name where the regression target (yield) will be stored.
        mod_data_point_role_column (str, optional): Column name indicating the role of each data point.
    """

    def __init__(
        self,
        spy_inf_train_data: pd.DataFrame,
        augmented_negatives: pd.DataFrame,
        known_positives: pd.DataFrame,
        random_state: int = 42,
        logger: Optional[Logger] = None,
        known_yield_column: str = None,
        augmented_target_column: Optional[str] = None,
        mod_data_point_role_column: Optional[str] = None
    ) -> None:
        """
        Initialize the DataRecombination class.

        Args:
            spy_inf_train_data (pd.DataFrame): Spy-infused training data.
            augmented_negatives (pd.DataFrame): DataFrame containing augmented negatives.
            known_positives (pd.DataFrame): DataFrame containing known positive examples.
            random_state (int, optional): Random seed for reproducibility. Default is 42.
            logger (Logger, optional): Logger instance for logging.
            known_yield_column (str, optional): Column name in known positives for true yield values. Default is "Output".
            augmented_target_column (str, optional): Column name for the regression target. Default is "meta_augmented_target".
            mod_data_point_role_column (str, optional): Column name indicating each data point's role. Default is "meta_mod_data_point_role".
        """
        self.spy_inf_train_data = spy_inf_train_data
        self.augmented_negatives = augmented_negatives
        self.known_positives = known_positives
        self.random_state = random_state
        self.logger = logger
        # Optional parameters for classification; if not provided, they can be set later.
        self.known_yield_column = known_yield_column
        self.augmented_target_column = augmented_target_column
        self.mod_data_point_role_column = mod_data_point_role_column

    @classmethod
    def from_config(
        cls,
        config: Dict[str, Any],
        spy_inf_train_data: pd.DataFrame,
        augmented_negatives: pd.DataFrame,
        known_positives: pd.DataFrame,
        logger: Optional[Logger] = None
    ) -> "DataRecombination":
        """
        Alternative constructor that extracts required parameters from a configuration dictionary.

        Args:
            config (Dict[str, Any]): Configuration dictionary.
            spy_inf_train_data (pd.DataFrame): Spy-infused training data.
            augmented_negatives (pd.DataFrame): DataFrame containing augmented negatives.
            known_positives (pd.DataFrame): DataFrame containing known positives.
            logger (Logger, optional): Logger instance.

        Returns:
            DataRecombination: A new instance configured from the provided config.
        """
        return cls(
            spy_inf_train_data=spy_inf_train_data,
            augmented_negatives=augmented_negatives,
            known_positives=known_positives,
            random_state=config["general"]["random_seed"],
            logger=logger,
            known_yield_column=config["dataset"]["target_column"],
            augmented_target_column=config["meta_columns"]["meta_augmented_target"],
            mod_data_point_role_column=config["meta_columns"]["meta_mod_data_point_role"]
        )


    def generate_augmented_negative_set(self, negative_yield_value: float, augmented_target_column_name: Optional[str] = None,
                                        true_regression_column_name: Optional[str] = None) -> pd.DataFrame:
        """
        Generate a regression dataset by combining known positives with augmented negatives.

        Known positives retain their true yield values (from known_yield_column),
        and augmented negatives are assigned the specified negative yield value.

        Args:
            negative_yield_value (float): Yield value to assign to augmented negatives.
            augmented_target_column_name (str, optional): Column name for the regression target.
            true_regression_column_name (str, optional): Column name in known positives for true yield values.

        Returns:
            pd.DataFrame: Combined dataset for training the regression model.
        """
        augmented_target_column_name = augmented_target_column_name or self.augmented_target_column
        true_regression_column_name = true_regression_column_name or self.known_yield_column

        known_pos = self.known_positives.copy()
        aug_neg = self.augmented_negatives.copy()

        # Map true yield to the target column for positives
        known_pos[augmented_target_column_name] = known_pos[true_regression_column_name]

        # Assign fixed negative yield value for negatives
        aug_neg[augmented_target_column_name] = negative_yield_value

        final_dataset = pd.concat([known_pos, aug_neg], ignore_index=False)
        if self.logger:
            self.logger._log_dataframe_as_artifact(final_dataset, "reg_augmented_negative_training_dataset.csv")

        validate_split_integrity([known_pos, aug_neg], [final_dataset])

        return final_dataset


    def generate_unlabeled_as_negative_set(self, negative_yield_value: float = 0.0,
                                           mod_data_point_role_column_name: Optional[str] = None,
                                           augmented_target_column_name: Optional[str] = None,
                                           true_regression_column_name: Optional[str] = None) -> pd.DataFrame:
        """
        Generate a regression dataset by treating unlabeled data as negatives.

        Known positives retain their true yield values, and unlabeled data (filtered based on mod_data_point_role_column)
        is assigned the specified negative yield value.

        Args:
            negative_yield_value (float, optional): Yield value to assign to unlabeled negatives. Defaults to 0.0.
            mod_data_point_role_column_name (str, optional): Override for the column name indicating data point role.
            augmented_target_column_name (str, optional): Override for the column name for regression target.
            true_regression_column_name (str, optional): Override for the column name for true regression values.

        Returns:
            pd.DataFrame: Combined dataset with noisy negatives for regression training.

        Raises:
            KeyError: If the expected role column is not found in spy_inf_train_data.
        """
        mod_data_point_role_column_name = mod_data_point_role_column_name or self.mod_data_point_role_column
        augmented_target_column_name = augmented_target_column_name or self.augmented_target_column
        true_regression_column_name = true_regression_column_name or self.known_yield_column


        known_pos = self.known_positives.copy()
        known_pos[augmented_target_column_name] = known_pos[true_regression_column_name]

        unlabeled_roles = ["unlabeled negative", "unlabeled positive"]
        unlabeled_dp = self.spy_inf_train_data.copy()
        if mod_data_point_role_column_name not in unlabeled_dp.columns:
            raise KeyError(f"Expected column '{mod_data_point_role_column_name}' not found in spy_inf_train_data.")
        unlabeled_dp = unlabeled_dp[unlabeled_dp[mod_data_point_role_column_name].isin(unlabeled_roles)].copy()

        # Assign fixed negative yield value for all unlabeled data (Naïve assumption)
        unlabeled_dp[augmented_target_column_name] = negative_yield_value

        final_dataset = pd.concat([known_pos, unlabeled_dp], ignore_index=False)
        if self.logger:
            self.logger._log_dataframe_as_artifact(final_dataset, "reg_unlabeled_as_negative_training_dataset.csv")

        validate_split_integrity([known_pos, unlabeled_dp], [final_dataset])

        return final_dataset

__init__(spy_inf_train_data, augmented_negatives, known_positives, random_state=42, logger=None, known_yield_column=None, augmented_target_column=None, mod_data_point_role_column=None)

Initialize the DataRecombination class.

Parameters:

Name Type Description Default
spy_inf_train_data DataFrame

Spy-infused training data.

 required
augmented_negatives DataFrame

DataFrame containing augmented negatives.

 required
known_positives DataFrame

DataFrame containing known positive examples.

 required
random_state int

Random seed for reproducibility. Default is 42.

 42
logger logging

Logger instance for logging.

 None
known_yield_column str

Column name in known positives for true yield values. Default is "Output".

 None
augmented_target_column str

Column name for the regression target. Default is "meta_augmented_target".

 None
mod_data_point_role_column str

Column name indicating each data point's role. Default is "meta_mod_data_point_role".

 None
Source code in payn\Recombination\datarecombination.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def __init__(
    self,
    spy_inf_train_data: pd.DataFrame,
    augmented_negatives: pd.DataFrame,
    known_positives: pd.DataFrame,
    random_state: int = 42,
    logger: Optional[Logger] = None,
    known_yield_column: str = None,
    augmented_target_column: Optional[str] = None,
    mod_data_point_role_column: Optional[str] = None
) -> None:
    """
    Initialize the DataRecombination class.

    Args:
        spy_inf_train_data (pd.DataFrame): Spy-infused training data.
        augmented_negatives (pd.DataFrame): DataFrame containing augmented negatives.
        known_positives (pd.DataFrame): DataFrame containing known positive examples.
        random_state (int, optional): Random seed for reproducibility. Default is 42.
        logger (Logger, optional): Logger instance for logging.
        known_yield_column (str, optional): Column name in known positives for true yield values. Default is "Output".
        augmented_target_column (str, optional): Column name for the regression target. Default is "meta_augmented_target".
        mod_data_point_role_column (str, optional): Column name indicating each data point's role. Default is "meta_mod_data_point_role".
    """
    self.spy_inf_train_data = spy_inf_train_data
    self.augmented_negatives = augmented_negatives
    self.known_positives = known_positives
    self.random_state = random_state
    self.logger = logger
    # Optional parameters for classification; if not provided, they can be set later.
    self.known_yield_column = known_yield_column
    self.augmented_target_column = augmented_target_column
    self.mod_data_point_role_column = mod_data_point_role_column

from_config(config, spy_inf_train_data, augmented_negatives, known_positives, logger=None) classmethod

Alternative constructor that extracts required parameters from a configuration dictionary.

Parameters:

Name Type Description Default
config Dict[str, Any]

Configuration dictionary.

 required
spy_inf_train_data DataFrame

Spy-infused training data.

 required
augmented_negatives DataFrame

DataFrame containing augmented negatives.

 required
known_positives DataFrame

DataFrame containing known positives.

 required
logger logging

Logger instance.

 None

Returns:

Name Type Description
DataRecombination DataRecombination

A new instance configured from the provided config.
Source code in payn\Recombination\datarecombination.py 
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
@classmethod
def from_config(
    cls,
    config: Dict[str, Any],
    spy_inf_train_data: pd.DataFrame,
    augmented_negatives: pd.DataFrame,
    known_positives: pd.DataFrame,
    logger: Optional[Logger] = None
) -> "DataRecombination":
    """
    Alternative constructor that extracts required parameters from a configuration dictionary.

    Args:
        config (Dict[str, Any]): Configuration dictionary.
        spy_inf_train_data (pd.DataFrame): Spy-infused training data.
        augmented_negatives (pd.DataFrame): DataFrame containing augmented negatives.
        known_positives (pd.DataFrame): DataFrame containing known positives.
        logger (Logger, optional): Logger instance.

    Returns:
        DataRecombination: A new instance configured from the provided config.
    """
    return cls(
        spy_inf_train_data=spy_inf_train_data,
        augmented_negatives=augmented_negatives,
        known_positives=known_positives,
        random_state=config["general"]["random_seed"],
        logger=logger,
        known_yield_column=config["dataset"]["target_column"],
        augmented_target_column=config["meta_columns"]["meta_augmented_target"],
        mod_data_point_role_column=config["meta_columns"]["meta_mod_data_point_role"]
    )

generate_augmented_negative_set(negative_yield_value, augmented_target_column_name=None, true_regression_column_name=None)

Generate a regression dataset by combining known positives with augmented negatives.

Known positives retain their true yield values (from known_yield_column), and augmented negatives are assigned the specified negative yield value.

Parameters:

Name Type Description Default
negative_yield_value float

Yield value to assign to augmented negatives.

 required
augmented_target_column_name str

Column name for the regression target.

 None
true_regression_column_name str

Column name in known positives for true yield values.

 None

Returns:

Type Description
DataFrame

pd.DataFrame: Combined dataset for training the regression model.
Source code in payn\Recombination\datarecombination.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
def generate_augmented_negative_set(self, negative_yield_value: float, augmented_target_column_name: Optional[str] = None,
                                    true_regression_column_name: Optional[str] = None) -> pd.DataFrame:
    """
    Generate a regression dataset by combining known positives with augmented negatives.

    Known positives retain their true yield values (from known_yield_column),
    and augmented negatives are assigned the specified negative yield value.

    Args:
        negative_yield_value (float): Yield value to assign to augmented negatives.
        augmented_target_column_name (str, optional): Column name for the regression target.
        true_regression_column_name (str, optional): Column name in known positives for true yield values.

    Returns:
        pd.DataFrame: Combined dataset for training the regression model.
    """
    augmented_target_column_name = augmented_target_column_name or self.augmented_target_column
    true_regression_column_name = true_regression_column_name or self.known_yield_column

    known_pos = self.known_positives.copy()
    aug_neg = self.augmented_negatives.copy()

    # Map true yield to the target column for positives
    known_pos[augmented_target_column_name] = known_pos[true_regression_column_name]

    # Assign fixed negative yield value for negatives
    aug_neg[augmented_target_column_name] = negative_yield_value

    final_dataset = pd.concat([known_pos, aug_neg], ignore_index=False)
    if self.logger:
        self.logger._log_dataframe_as_artifact(final_dataset, "reg_augmented_negative_training_dataset.csv")

    validate_split_integrity([known_pos, aug_neg], [final_dataset])

    return final_dataset

generate_unlabeled_as_negative_set(negative_yield_value=0.0, mod_data_point_role_column_name=None, augmented_target_column_name=None, true_regression_column_name=None)

Generate a regression dataset by treating unlabeled data as negatives.

Known positives retain their true yield values, and unlabeled data (filtered based on mod_data_point_role_column) is assigned the specified negative yield value.

Parameters:

Name Type Description Default
negative_yield_value float

Yield value to assign to unlabeled negatives. Defaults to 0.0.

 0.0
mod_data_point_role_column_name str

Override for the column name indicating data point role.

 None
augmented_target_column_name str

Override for the column name for regression target.

 None
true_regression_column_name str

Override for the column name for true regression values.

 None

Returns:

Type Description
DataFrame

pd.DataFrame: Combined dataset with noisy negatives for regression training.

Raises:

Type Description
KeyError

If the expected role column is not found in spy_inf_train_data.
Source code in payn\Recombination\datarecombination.py 
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def generate_unlabeled_as_negative_set(self, negative_yield_value: float = 0.0,
                                       mod_data_point_role_column_name: Optional[str] = None,
                                       augmented_target_column_name: Optional[str] = None,
                                       true_regression_column_name: Optional[str] = None) -> pd.DataFrame:
    """
    Generate a regression dataset by treating unlabeled data as negatives.

    Known positives retain their true yield values, and unlabeled data (filtered based on mod_data_point_role_column)
    is assigned the specified negative yield value.

    Args:
        negative_yield_value (float, optional): Yield value to assign to unlabeled negatives. Defaults to 0.0.
        mod_data_point_role_column_name (str, optional): Override for the column name indicating data point role.
        augmented_target_column_name (str, optional): Override for the column name for regression target.
        true_regression_column_name (str, optional): Override for the column name for true regression values.

    Returns:
        pd.DataFrame: Combined dataset with noisy negatives for regression training.

    Raises:
        KeyError: If the expected role column is not found in spy_inf_train_data.
    """
    mod_data_point_role_column_name = mod_data_point_role_column_name or self.mod_data_point_role_column
    augmented_target_column_name = augmented_target_column_name or self.augmented_target_column
    true_regression_column_name = true_regression_column_name or self.known_yield_column


    known_pos = self.known_positives.copy()
    known_pos[augmented_target_column_name] = known_pos[true_regression_column_name]

    unlabeled_roles = ["unlabeled negative", "unlabeled positive"]
    unlabeled_dp = self.spy_inf_train_data.copy()
    if mod_data_point_role_column_name not in unlabeled_dp.columns:
        raise KeyError(f"Expected column '{mod_data_point_role_column_name}' not found in spy_inf_train_data.")
    unlabeled_dp = unlabeled_dp[unlabeled_dp[mod_data_point_role_column_name].isin(unlabeled_roles)].copy()

    # Assign fixed negative yield value for all unlabeled data (Naïve assumption)
    unlabeled_dp[augmented_target_column_name] = negative_yield_value

    final_dataset = pd.concat([known_pos, unlabeled_dp], ignore_index=False)
    if self.logger:
        self.logger._log_dataframe_as_artifact(final_dataset, "reg_unlabeled_as_negative_training_dataset.csv")

    validate_split_integrity([known_pos, unlabeled_dp], [final_dataset])

    return final_dataset