deepword.students package¶
Submodules¶
deepword.students.snn_learner module¶
-
class
deepword.students.snn_learner.SNNData(target_src, same_src, diff_src)¶
-
class
deepword.students.snn_learner.SNNLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶ Bases:
deepword.students.student_learner.StudentLearner-
get_snn_tjs(tjs: deepword.trajectory.Trajectory, tid_sid_set: List[Tuple[int, int]]) → Tuple[List[List[int]], List[int], List[List[int]], List[List[deepword.agents.utils.ActionMaster]]]¶
-
preprocess_input()¶
-
snn_data_loader(data_path: str, batch_size: int, training: bool) → Generator[deepword.students.snn_learner.SNNData, None, None]¶
-
test(device_placement: str = '/device:GPU:0', restore_from: Optional[str] = None) → Tuple[int, int]¶
-
train(n_epochs: int) → None¶
-
deepword.students.student_learner module¶
-
class
deepword.students.student_learner.DRRNLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: str)¶
-
class
deepword.students.student_learner.GenConcatActionsLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶ Bases:
deepword.students.student_learner.GenLearner- TODO: we choose action_mask in memory; however, when all admissible actions
are required, e.g. in pre-training without using q-values, we can use sys_action_mask. Notice that sys_action_mask won’t compatible with q-values, since the q-values are computed against action_mask.
-
classmethod
concat_actions(action_ids: List[numpy.ndarray], action_len: List[int], action_weight: numpy.ndarray, sep_val_id: int, sort_by_weight: bool = True) → Tuple[numpy.ndarray, numpy.ndarray]¶ concat given action_ids into one output str
-
class
deepword.students.student_learner.GenLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶
-
class
deepword.students.student_learner.GenMixActionsLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶
-
class
deepword.students.student_learner.NLUClassificationLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶
-
class
deepword.students.student_learner.NLULearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶
-
class
deepword.students.student_learner.StudentLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶ Bases:
deepword.log.Logging-
classmethod
hs2tj_old2new(old_hs2tj: Dict[str, Dict[int, List[int]]]) → Dict[str, Dict[int, List[int]]]¶ sid need to be halved. :param old_hs2tj:
Returns:
-
classmethod
lst_str2am(tj: List[str], allow_unfinished_tj: bool = False) → List[deepword.agents.utils.ActionMaster]¶
-
classmethod
memo_old2new(old_memo: List[deeptextworld.agents.base_agent.DRRNMemoTeacher]) → List[deepword.agents.utils.Memolet]¶
-
safe_loading(model: Any, sess: tensorflow.python.client.session.Session, saver: tensorflow.python.training.saver.Saver, restore_from: str) → int¶ Load weights from restore_from to model. If weights in loaded model are incompatible with current model, try to load those weights that have the same name.
This method is useful when saved model lacks of training part, e.g. Adam optimizer.
- Parameters
model – A tensorflow model
sess – A tensorflow session
saver – A tensorflow saver
restore_from – the path to restore the model
- Returns
training steps
-
test() → None¶
-
classmethod
tjs_str2am(old_tjs: deepword.trajectory.Trajectory[str]) → deepword.trajectory.Trajectory[deepword.agents.utils.ActionMaster]¶
-
train(n_epochs: int) → None¶
-
classmethod
deepword.students.swag_learner module¶
-
class
deepword.students.swag_learner.SwagLearner(hp: tensorflow.contrib.training.python.training.hparam.HParams, model_dir: str, train_data_dir: Optional[str], eval_data_path: Optional[str] = None)¶ Bases:
deepword.students.student_learner.NLUClassificationLearner-
test(device_placement: str = '/device:GPU:0', restore_from: Optional[str] = None) → Tuple[int, int]¶
-
-
deepword.students.swag_learner.get_bert_input(start_str: List[str], ending_str: List[numpy.ndarray], tokenizer: deepword.tokenizers.Tokenizer, sep_val_id, cls_val_id, num_tokens, n_tokens_per_action) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]¶ TODO: here we use n_tokens_per_actions for end str size
-
deepword.students.swag_learner.load_swag_data(fn_swag: str) → Tuple[List[str], List[numpy.ndarray], List[int]]¶ read swag data, make sure every start sentence has four ends
-
deepword.students.swag_learner.process_one_line(tokenizer: deepword.tokenizers.Tokenizer, start_str: str, ending_str: List[str], max_start_len: int, max_end_len: int) → Tuple[List[int], int, numpy.ndarray, numpy.ndarray]¶
deepword.students.utils module¶
-
deepword.students.utils.align_batch_str(ids: List[List[int]], str_len_allowance: int, padding_val_id: int, valid_len: List[int]) → Tuple[numpy.ndarray, numpy.ndarray]¶ Align a batch of string indexes. The maximum length will not exceed str_len_allowance. Each array of ids will be either padded or trimmed to reach the maximum length, notice that padded length won’t be counted in as valid length.
- Parameters
ids – a list of array of index (int)
str_len_allowance –
padding_val_id –
valid_len –
- Returns
aligned ids and aligned length
-
deepword.students.utils.batch_dqn_input(trajectories: List[List[deepword.agents.utils.ActionMaster]], tokenizer: deepword.tokenizers.Tokenizer, num_tokens: int, padding_val_id: int, with_action_padding: bool = False, max_action_size: Optional[int] = None) → Tuple[List[List[int]], List[int], List[List[int]]]¶
-
deepword.students.utils.dqn_input(trajectory: List[deepword.agents.utils.ActionMaster], tokenizer: deepword.tokenizers.Tokenizer, num_tokens: int, padding_val_id: int, with_action_padding: bool = False, max_action_size: Optional[int] = None) → Tuple[List[int], int, List[int]]¶ Given a trajectory (a list of ActionMaster), get trajectory indexes, length and master mask (master marked as 1 while action marked as 0). Pad the trajectory to num_tokens. Pad actions if required.
-
deepword.students.utils.tj2ids(trajectory: List[deepword.agents.utils.ActionMaster], tokenizer: deepword.tokenizers.Tokenizer, with_action_padding: bool = False, max_action_size: Optional[int] = None, padding_val_id: int = 0) → Tuple[List[int], List[int]]¶ Convert a trajectory (list of ActionMaster) into ids Compute segmentation ids for masters (1) and actions (0) pad actions if required.