Frequent assessment of the severity of illness for hospitalized patients is essential in clinical settings to prevent outcomes such as in-hospital mortality and unplanned ICU admission. Classical severity scores have been developed typically using relatively few patient features, especially for intensive care. Recently, deep learning-based models demonstrated better individualized risk assessments compared to classic risk scores such as SOFA and NEWS, thanks to the use of aggregated and more heterogeneous data sources for dynamic risk prediction. We investigated to what extent deep learning methods can capture patterns of longitudinal change in health status using time-stamped data from electronic health records. We used medical history data, biochemical measurements, and the clinical notes from all patients admitted to non-intensive care units in 12 hospitals in Denmarks Capital Region and Region Zealand during 2011-2016. Data from a total of 852,620 patients and 2,241,849 admissions were used to predict the composite outcome of unplanned ICU transfer and in-hospital death at different time points after admission to general departments. We subsequently examined feature interpretations of the models. The best model used all data modalities with an assessment rate of 6 hours and a prediction window of 14 days, with an AUPRC of 0.287 and AUROC of 0.898. These performances are comparable to the current state of the art and make the model suitable for further prospective validation as a risk assessment tool in a clinical setting.